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ABSTRACT 

As colleges and universities nationwide anticipate enrolling more than two million new students over 
the next decade, UC Berkeley is exploring options for serving more students, more cost effectively, in 
large lecture courses. This research project analyzes economic and pedagogical questions related to the 
use of on-line lecture and laboratory material in a large introductory chemistry course at UC Berkeley. 
We undertook a quasi-experimental two-year study to determine if the utilization of on-line teaching 
materials results in significant restructuring of staff time in laboratories and lectures, if teaching 
facilities can be used by more students, and if the technology enhancements affect student performance 
and/or attitudes. What emerged is a rich, yet complicated, profile of the effects that technology 
enhancements have on the individuals and organizations involved in implementation and testing. 
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EXECUTIVE SUMMARY 

BACKGROUND 

Public and private colleges and universities nationwide expect to enroll more than two million new full-time 
students by 2010, a phenomenon referred to as Tidal Wave II. The University of California (UC) ten-campus 
system faces an increased enrollment of almost 63,000 lull-time students — a 43% increase. The anticipated 
influx of new students over the next decade has prompted UC Berkeley to explore options for serving more 
students, more cost effectively, without increasing teaching and support staff in large lecture courses. It has 
been argued that the strategic use of on-line resources in large lecture classes can result in some savings and 
redistribution of teaching staff time, also known as a substitution of capital for labor. Determining the 
effectiveness of technology enhancements in higher education settings, however, is not a simple undertaking. 

This Andrew W. Mellon Foundation-funded research project analyzes economic and pedagogical questions 
related to the use of on-line lecture and laboratory material in a large introductory chemistry course at 
UC Berkeley. We undertook a quasi-experimental two-year study of this course to determine if: 

• the utilization of on-line teaching materials results in significant restructuring of staff time in laboratories 
and lectures, 

• teaching facilities could be used by more students, and 

• the technology enhancements affect student performance and/or attitudes. 

A primary goal of this research was to place our findings within the larger context of the institution. Therefore 
we used a wide range of data collection techniques to track student and staff behavior, economic costs, and 
campus culture. What emerged is a rich, yet complicated, profile of the effects that technology enhancements 
have on the individuals and organizations involved in implementation and testing. 

STUDY DESIGN 

Chemistry 1 A, the first-semester introductory chemistry course offered at UC Berkeley, has one of the largest 
enrollments of any course taught on campus. Approximately 2,000 students take this introductory course each 
year, and approximately 100 teaching and support staff are required to teach and manage the course. The 
technologically-enhanced version of the course is referred to as “Digital Chemistry 1 A.” The technology 
enhancements in this course included: 

• deployment of on-line quizzes and pre-laboratory assignments; 

• conversion of the lecture chalkboard content to digital slides; and, 

• broadcast of video lectures, with synchronized and indexed slides, over the Internet for on-demand replay. 

Our research was conducted over two academic years: 2000-2001 (Year 1) and 2001-2002 (Year 2). We 
collected a wide range of quantitative and qualitative data on student performance and attitudes and on 
faculty/staff teaching and preparation activities. As a part of our cost analysis in Year 1, we conducted a 
controlled experiment between students who did and did not have access to selected technology enhancements. 
In Y ear 2, the Department of Chemistry decided that all technology enhancements would be made available to 
all students, and we were unable to replicate a similar quasi-experimental design. 

Measuring Cost Effectiveness. Our goal was to compare overall course costs for the two formats of instruction 
in Chemistry 1A (traditional and technology-enhanced) using an activity-based costing method. We identified 
activities through interviews of the instructors, Teaching Assistants (TAs), and non-teaching staff who were 
involved with the course. We also observed lectures and labs. We measured the cost of resources used, which 
are not necessarily the same as the resources acquired by the university for the course. Activities for the course 
were divided into development, delivery, and revision categories. 
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Teaching Staff Time and Attitudes. We were interested in redistribution of staff time; therefore, we used a 
variety of methods to collect data on teaching staff (instructors and TAs) time and attitudes over the two-year 
study. These methods included interviews, student and TA surveys, time logs, observations, and focus groups. 

Student Background, Performance, and Retention. We examined the impact of the technology 
enhancements on various aspects of student performance: student learning as measured by grades on quizzes 
and exams, a carry-forward experiment, and course retention rates. We also analyzed performance relative to 
student demographic and other background data. 

Student Attitudes about the Course. We used a combination of pre- and post-surveys and focus groups to 
measure other possible changes in student learning, such as student access to the technology used in the course, 
their use of it, their opinions regarding using it, whether or not the students believe it impacted their learning, 
and how it affected their attitudes towards the course and learning. 

Student Use of Technology. We collected usage statistics for both years of the study. Analysis of usage 
statistics included these on-line features: lecture webcasts, slide presentations from lecture, quizzes, lab manual, 
course information and homework assignments. 

Other Courses. For comparative purposes, we collected and analyzed a variety of data from other chemistry 
and non-chemistry courses on campus. These data included randomized visual attendance scans, student 
evaluations, and faculty interviews. 

DISCUSSION OF FINDINGS 

Although the technology enhancements increased the cost of Chemistry 1A in the initial pilot year, these 
costs are a relatively small percentage of the total cost of the course. The technology-enhanced course in 
Year 1 cost $13 (1.7%) more per student than the traditional course. In Year 2, the technology-enhanced course 
cost $59 (7.8%) less per student than the traditional course. The investments in technology-enhanced materials 
(development) paid for themselves over the two course offerings (one offering each in Y ears 1 and 2) when 
savings in delivery costs are calculated in. 

Development costs (website, lecture slides, on-line quizzes) would decrease in future years if instructors 
were to revise or reuse existing digital or multimedia products in their courses. The degree to which reuse 
of the technology enhancements by other faculty will occur is not clear, as the introductory chemistry course at 
UC Berkeley is taught on a rotating basis by tenure track faculty who are active researchers. The result is that 
each faculty member has a distinct philosophy, strong preferences, and considerable flexibility in how to teach 
the class. 

Instructors spend less time doing repetitive tasks in the technology-enhanced version of Chemistry 1A. 

Specifically, our data show that instructors spend considerably less time preparing for class since the 
introduction of the lecture slides. 

Instructors spend less time answering routine questions in the technology-enhanced course because 
students are able to find the necessary information on-line. Instructors spend approximately 50% less time 
answering routine questions about the course, including time spent in office hours. More than 60% of students 
report that they go to the website rather than teaching staff office hours to get answers to questions at least some 
of the time. 

Teaching Assistants are relatively inexperienced teachers and spend a large amount of their time at the 
start of the semester negotiating the varied responsibilities of being a TA, not using technology to enhance 
their teaching. TA surveys indicate that that by some measures, they were more comfortable with the 
technologies and the benefits provided as the semester progressed (e.g., use of webcasts, perception that 
technologies saved time and freed up time in lab). 
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The TAs in the treatment group spent less time grading, and appeared to spend less time on 
administrative tasks both in and out of the classroom. The availability of on-line grading of quizzes reduced 
time spent grading, which is a task most TAs find menial. Because TA salaries and benefits are 60% of all 
course costs for Chemistry 1A, reducing, or at least reallocating, TA time presents opportunities for saving 
money, serving more students, and/or redistributing TA time to allow for richer interactions with students. 

Laboratory sections could hypothetically be reduced from four hours to three to better utilize lab space. 

If a time reduction proves practical, Chemistry 1A could add approximately 20 lab sections per week and 
accommodate approximately 600 students without acquiring new space for labs. While more TAs would need 
to be hired to teach additional sections, no additional costs would be incurred for new facilities in this scenario. 
Although data show that a reduction of lab time from four to three hours is possible, it is not probable. The 
four-hour section seems to be the desired interval for the activities that take place in lab, which include not only 
the experiment but formal discussions and informal one-on-one interaction among students and TAs. 

Student performance was not significantly affected by the technology enhancements in the Year 1 
experiment. We have found no significant difference between students in the treatment and control groups in 
grades, retention, or their conceptual understanding in the following semester of chemistry. 

Students find the technologies to be an exceptionally positive component of the course. A little explored 
topic in cost-effectiveness studies is the impact on student “costs” (i.e., what do students perceive as 
benefits/costs of the technology?). Our attitudinal data collected over two years suggest that students perceived 
the suite of enhancements as a significant contributor to their overall satisfaction with this large lecture course, 
web usage data, when triangulated with performance and attitudinal data in Y ear 2, suggest that students use the 
enhancements: (a) on an “as needed” basis; (b) as a significant resource in their study strategies, especially when 
preparing for exams; and (c) as safety nets for their individual circumstances (e.g., disabilities, English 
proficiency, personal schedules). 

Lectures can be a positive draw for students. Our findings from Chemistry 1A and an introductory 
astronomy course show that excellent lectures presented by dynamic teaching staff are a huge draw for students. 
In Chemistry 1A, reasons for attending the lectures included interaction with other students and the instructors, 
the experience of live demonstrations, and improved personal discipline and concentration. 

A large number of students regularly do not attend lectures. At no time was full lecture hall capacity 
(N=1569) approached in our attendance counts (range=762 to 1024). In Year 2, 31% of survey respondents 
report attending lecture less than three times per week and 25% report replacing the lecture with webcasts. 
Attendance data on another introductory science course, which did not use webcasts, indicate that webcasts 
alone are not the reason for decreased student attendance at lectures. Comparative attendance and viewing data 
from other courses that used on-line video lecture archives at UC Berkeley suggest that the degree to which 
students opt out of attending lectures may be heavily influenced by time of day and style of lecture delivery. 

The availability of the on-demand replays of lectures has the potential to allow a larger number of 
students to be enrolled in the course. Our data indicate that most students in Chemistry 1A use the on-line 
lecture webcasts primarily as study aids, and the majority (>80%) would not substitute remote viewing for 
attending lecture. Students still report, and we observed, however, that they do not attend lecture the “required” 
three days per week — but rather closer to an average of two days per week. 

Reduction in the number of lectures given each day from three to two (or one) — perhaps by requiring 
some students to attend lectures virtually — could realize appreciable savings in faculty time devoted to 
lecture as well as free up lecture hall space for other courses. Apparently all students attending lectures 
could have been accommodated in two lectures instead of the three that were allocated. Because the same 
lecture is given three times per day, staff and facilities costs could be saved if a proportion of students formally 
opted out of attending lectures. 
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CONCLUSION 

Challenges to Conducting Robust Research 

The challenges associated with executing a robust research analysis of a “fast-running” experiment of this scope 
are substantial. The size and complexity of the Chemistry 1A teaching and learning environment and its 
placement within an even larger and more complex public research university cannot be overemphasized. 
Implementation and evaluation of large-scale experiments of this sort require not only robust campus technology 
support structures, but the gathering of different types of data (costs, learning outcomes, server statistics) from 
disparate campus units and individuals (institutional, faculty, staff, students, etc.). 

The Importance of Convenience and Choice for Students 

Large lecture courses have a reputation among educators as being poor learning settings, although there are 
good data that suggest lectures serve many useful purposes for students and faculty. Our data show, however, 
that students were both exceptionally enthusiastic about the lecture component of the course and engaged with 
the on-line materials. Survey responses and transaction log analysis showed that the course website in general, 
and the lecture slides posted on the website in particular, were popular and well-received. Transaction log 
analysis of lecture webcasts showed clearly how and why students used this on-line resource. Attendance data 
indicate that, although students valued lectures, they frequently opted out of attending them. 

We posit that the positive reception of the Chemistry 1 A course and the associated technology enhancements is 
related to a number of factors, which include the fact that the enhancements were minimally disruptive to the 
teaching style and pedagogy of the teaching staff, they increased convenience for both students and faculty, and 
they were “generic” and pedagogically neutral enough that students could use them flexibly and on their own 
terms. 

The Implications for Sharing, Reuse, and University Culture Change 

Campus culture will have a significant impact on the likelihood that on-line teaching materials will be shared 
and reused by other faculty. Our findings suggest that some cost-savings could be realized under certain 
circumstances, which may or may not carry over from semester to semester at UC Berkeley or other campuses 
that pride themselves on having active research faculty teach introductory courses. Our knowledge of 
Chemistry 1 A and other campus faculty behavior suggests that the successful wholesale adoption of technology 
enhancements from one semester to the next cannot be assumed. Replicating support mechanisms and 
customizing materials to one’s own course require investments of time and energy by teaching staff. We should 
note that the experience at UC Berkeley is probably not directly comparable to institutions where non-research 
faculty are responsible for teaching large introductory courses. In fact, the sharing of electronic teaching 
materials among faculty may occur more readily in institutions where introductory course curricula are 
standardized and where research faculty cede course development and delivery to lecturers or adjuncts. 

Given a change in campus culture and thinking, there is certainly the possibility that several UC Berkeley 
instructors, or even instructors on other UC campuses, might be able to share on-line lecture materials. 
Additionally, a rethinking of the time faculty devote to repetition of the same lectures multiple times in a week 
could potentially free instructors to creatively use the lecture time as a more student-interactive experience 
and/or reallocate space for other purposes. This rethinking seems particularly relevant given that students have 
independently found ways to integrate technology enhancements into their time management and study 
strategies. 

Finally, we suspect that any large scaling benefits will come either (a) when newly hired faculty, who might be 
more adroit with new technologies, enter the department; (b) if the course can be “modular” so that faculty can 
select materials that fit their learning goals, should their learning goals differ from the developers’ intentions; 
and/or (c) if the materials can be made available to off-site student populations at other institutions. 
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Costs, Culture, and Complexity: An Analysis of Technology 
Enhancements in a Large Lecture Course at UC Berkeley 1 

I. INTRODUCTION 

Public and private colleges and universities nationwide expect to enroll more than two million new full-time 
students by 2010, a phenomenon referred to as Tidal Wave II (CPEC, 2000). The University of California (UC) 
ten-campus system faces an increased enrollment of almost 63,000 full-time students — a 43 percent increase. 
The University of California, Berkeley (UC Berkeley) campus is being asked to explore how to absorb an 
additional 4,000 students by 2010. The anticipated influx of new students over the next decade has prompted 
UC Berkeley to explore options for serving more students, more cost effectively, without increasing teaching 
and support staff in large lecture courses. As with other campuses, UC Berkeley is contemplating a range of 
solutions that includes offering classes during the summer, expanding regular enrollments during fall and spring 
semesters, and making use of technology to expand on- and off-campus learning opportunities. 

It has been argued that the strategic use of online resources in large lecture classes can result in some savings 
and redistribution of teaching staff time, also known as a substitution of capital for labor (Massy & Zemsky, 
1995; Twigg, 2003). Determining the effectiveness of technology enhancements in higher education settings, 
however, is not a simple undertaking (see, for example, Phipps & Merisotis, 1999, and Fisher & Nygren, 2000). 
This paper reports on a rigorous economic and pedagogical analysis of questions related to the use of online 
lecture and laboratory material in an online introductory science course, and their potential to free up teaching 
staff time and/or serve more students off-site. Our primary goals were to determine: 

• if the utilization of online teaching materials results in significant restructuring of staff time in laboratories 
and lectures, 

• if teaching facilities can be used by more students, and 

• if the technology enhancements affect student performance and/or attitudes. 

To answer these questions, we undertook a quasi-experimental two-year study (September 2000 to June 2002) 
of the use of technology enhancements in the teaching of Chemistry 1A. A primary goal of this study was to 
place our findings within the larger context of the institution. Therefore we used a wide range of data collection 
techniques to track student and staff behavior, economic costs, and campus culture. What emerged is a rich, yet 
complicated, profile of the effects that technology enhancements have on the individuals and organizations 
involved in implementation and testing. 



1 This work is supported by a grant from the Andrew W. Mellon Foundation's Cost Effective Uses of Technology in Teaching (CEUTT) 
program initiative. Additional support was provided by UC Berkeley’s Center for Studies in Higher Education, the Berkeley Multimedia 
Research Center, the College of Chemistry, and the University of California, Berkeley. 
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II. BACKGROUND 

A. Course Description 

Chemistry 1 A is one of the largest, most visible courses at UC Berkeley — nearly 2,000 students, or one half of 
the freshman class, enroll in Chemistry 1A each year 2 and approximately 100 teaching and support staff are 
required to teach and manage the course. In addition to the large number of students served and the large 
number of staff involved in the course, Chemistry 1 A is also an important gateway to more advanced study in 
many disciplines. The technology enhancements in the UCB “Digital Chemistry 1A” course include: 

• deployment of online quizzes and pre-laboratory assignments; 

• conversion of the lecture chalkboard content to PowerPoint slides; and, 

• broadcast of video lectures, with synchronized and indexed slides, over the Internet for on-demand replay. 

Table 1 summarizes the course and its components. In the fall semester, one of three semesters in which the 
course is taught, one to two instructors (tenured faculty and/or lecturers) deliver nine lectures (three lectures per 
day, three days per week). These instructors are supported by approximately 50 Teaching Assistants (TAs), of 
whom 45-50 teach lab sections. An additional eight support staff also play an active role in implementing 
Chemistry 1A by assisting with lecture demonstrations, coordinating lab sections, preparing laboratory rooms 
for student experiments, and other miscellaneous tasks. 

Table 1: Overview of the Course 





Year 1 


Year 2 






Fall 


Fall 




Category 


semester 


semester 


Description 


Enrollment 


1,258 


1,202 


Students enrolled after third week. 


Students in Treatment Group 


287 


n/a 


Students enrolled after third week. 


Students in Control Group 


971 


n/a 


Students enrolled after third week. 


Lectures 


9 


9 


3 different one-hour lectures presented 3 times (MWF) per week; 
lecture delivered 3 times per day in 523-seat lecture hall. 


Lab Sections (Total) 


45 


42 


Per week. Each section met once per week, allotting 1 hour to 
discussion, 3 hours to experiment; up to 12 four-hour sections were 
scheduled daily Mon-Fri. 


Lab Sections (Treatment) 


11 


n/a 


Per week. Students in these sections had access to on-line quizzes 


Lab Sections (Control) 


34 


n/a 


plus all other technology. 

Per week. Students in these sections did not have access to on-line 
quizzes but did have access to all other technologies. 


Lab Section Meetings 


14 


14 


Per semester per lab section. Students are required to attend 1 
section per week. A total of 630 lab section meetings (45x14) took 
place during the Year 1 fall semester (588 meetings in Year 2) 


Experiments during section 


10 


10 


Per semester per lab section. 


Exam review during section 


4 


4 


Per semester per lab section. 


Exams 


4 


4 


Per semester. 3 two-hour midterms and 1 three-hour final exam are 
given each semester. 


Exam Proctoring & Grading 
TA Meetings 
Office Hours 


~32 hrs 
16.5 hrs 
30 hrs 


~32 hrs 
16.5 hrs 
30 hrs 


Hours per semester per TA (~7-9 hours per exam) 
Hours per semester per TA (1.5 hours per week) 
Hours per semester per TA (2 hours per wk) 


STAFF 

Teaching Staff — Instructors 
Teaching Staff — TAs 


2 

49 


2 

45 


1 professor and 1 lecturer 

Total TAs employed for the course @ 20 hours/week, salaried 


Lab section TAs 


45 


42 


TAs who led lab sections 


Head TAs 


3 


2 


TAs who organized curriculum, course policies 


E-TA 


1 


1 


TA who assisted with on-line technologies 


Non-teaching Staff 


~33 


~33 




Permanent 


8 


8 


Lab Manager, Stockroom Manager, Demonstration Expert, 
Webmaster, and other administrative staff 


Temporary 


~25 


~25 


Part-time staff and student employees 



2 During both Year 1 and Year 2 freshmen comprised approximately 89 percent of Chemistry 1A. 
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B. Study Design 

Determining the effectiveness of technology enhancements in higher education has been an elusive exercise (see 
for example, Phipps and Merisotis, 1999; Fisher and Nygren, 2000). The Mellon Foundation's grant allowed us 
to experiment with a wide range of data collection methods. Our goal was to employ as many techniques as 
possible to maximize our ability to triangulate findings within a highly complex organizational, cultural, and 
technological environment. 

The study was conducted over two academic years: 2000-2001 (Year 1) and 2001-2002 (Year 2). As a part of 
our cost analysis in Year 1, we conducted a controlled experiment between students who did and did not have 
access to selected technology enhancements. In Year 2, the Department of Chemistry decided that all 
technology enhancements would be made available to all students, and we were unable to replicate a similar 
quasi-experimental design. 

In Year 1, students and TAs were divided into two groups, with differing access to specific technology 
enhancements. The course content and requirements for each group were identical; the only difference was the 
medium used to accomplish certain tasks. Of the 45 scheduled lab sections, we randomly assigned students and 
TAs in 1 1 sections to the treatment group; those in the remaining 34 sections were in the control group. Of the 
total students enrolled in Chemistry 1A during Year 1, 23 percent of students (287) were in the treatment group 
and 77 percent of students (971) were in the control group. Students and TAs could not opt in or out of the two 
groups. The treatment group required students to perform homework quizzes and pre-laboratory assignments 
on-line — tasks that students in the control group conducted in labs. Table 2 summarizes the types of technology 
available to students in the two groups. All students had access to all technologies in Year 2. 

Table 2: Student Access to Technology Enhancements, Control vs. Treatment Groups 



Technology Enhancement 


Control Group 


Year 1 

Treatment Group 


Year 2 
All students 


Lecture 








Lecture slides 


X 


X 


X 


Lecture Webcasts 


X 


X 


X 


Course Website 


X 


X 


X 


Lab 








On-line Bulletin Board 


X 


X 


X 


On-line Lab Manual 


X 


X 


X 


On-line Pre-lab Assignment 


n/a 


X 


X 


On-line Homework Quiz 


n/a 


X 


X 


On-line Office Hours 


n/a 


n/a 


X 



C. Technology Enhancements 

It is important to note that Chemistry 1A is constantly changing and evolving; technology enhancements have 
been introduced to the course in an incremental fashion over the span of several years. Therefore we were 
unable to directly observe a fully traditional or fully technology-enhanced course during Y ear 1 . In order to 
draw comparisons of costs between the traditional and technology-enhanced versions of the course, it was 
necessary to define a hypothetical traditional course, even though we were unable to observe a fully traditional 
course. We did this with the aid of retrospective interviews of instructors and other staff. 

Table 3 provides an overview of the technologies making up three “versions” of the course: Traditional, 
technology-enhanced Year 1, and technology-enhanced Year 2. We distinguished between traditional and 
technology-enhanced costs only during Year 1 of the study. During Year 2, the course was fully technology 
enhanced and did not include a control group. 
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Table 3: Traditional versus Technology-Enhanced Version of the Course 



Traditional 

Hypothetical Construct 


Technology-enhanced 
Year 1: Hybrid Experiment, Differential 
Access 


Technology-enhanced 
Year 2: All students have access 


Chalkboards 


Digital slide presentations (PowerPoint) 


Digital slide presentations (PowerPoint) 


(No lecture webcasts) 


Webcasts of lecture 


Webcasts of lecture 


In-class video projection and videotaping 


In-class video projection and videotaping 


In-class video projection and videotaping 


Hard copies of pre-lab assignments turned 
in to and graded by TA. 


* On-line pre-lab assignments — graded 
automatically 


On-line pre-lab assignments — graded 
automatically 


In-class homework quizzes — graded by 
TA 


* On-line homework quizzes — graded 
automatically 


On-line homework quizzes — graded 
automatically 


(No course website) 


Course website 


Course website 


(No on-line grades database) 


On-line grades database 


On-line grades database 


(No course materials available on-line) 


On-line course materials (course syllabus, 
lab manual, etc.) 


On-line course materials (course syllabus, 
lab manual, etc.) 



* Only the treatment group had access to these technologies. All other technologies were available to the entire class. 



D. Study Participants 

Staff (teaching and non-teaching staff) and students were invited to participate in this study by members of the 
research team. Though students and TAs could not opt out of the type of section in which they were enrolled 
(treatment or control group), we were required by the Committee for the Protection of Human Subjects (CPHS) 
to get student consent to collect individual data. 3 Table 4 shows the number of students completing the course 
who consented to participate. 

Table 4: Number of Consenting Students Who Completed the Course 





Year 1 i 


| Year 2 | 




Enrolled 


j Consented 1 


Enrolled 


Consented 




% 


N 


% 


N 


% N 


% N 


Total number of students 


100% 


1,190 


56% 


672 


100% 1,135 


70% 797 


Students in Treatment Group 


23% 


277 


61% 


168 


n/a 


n/a 


Students in Control Group 


77% 


913 


55% 


504 


n/a 


n/a 



3 In Year 1, we did not collect data on minors because it was too onerous to collect parental consent forms; however, in Year 2, we were 
able to collect this infonnation because CPHS waived the parental consent requirement based on the fact that most students turned 18 
during the fall semester. This resulted in higher response rates in Year 2. 
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III. RESULTS 

A. Measuring Cost Effectiveness 

Our goal was to compare overall course costs for the two formats of instruction in Chemistry 1 A (traditional and 
technology-enhanced). As recommended by Levin and McEwan (2001) and Erhmann and Milam (1999), we 
estimated the cost of resources used to teach Chemistry 1A using activity-based costing. 4 

To identify activities used in either the traditional or technology-enhanced version of Chemistry 1A, we 
interviewed instructors, TAs and non-teaching staff who were involved with the course, and we observed 
lectures and labs. Based on these interviews and observations, we identified 50 activities necessary for offering 
Chemistry 1 A, which we list in Appendix A. For each activity, we collected cost data related to the staff, 
supplies and equipment, and facilities required to perform that activity. We collected data about both ongoing 
course-delivery activities and activities related to the development and revision of course materials. 

We measured the cost of resources used, which are not necessarily the same as the resources acquired by the 
university for the course. For example, we find that TAs use less time for particular grading activities in the 
technology-enhanced version of the course. We view such a reduction in time as a reduction in the cost of 
resources used for those grading activities regardless of whether the university reduces its TA payroll or 
reassigns the teaching assistants to other productive tasks or does nothing with the freed up TA time. 5 

1. Data Collection 

la. Interviews 

We interviewed the instructors as well as non-teaching staff who were involved in the coordination and teaching 
of Chemistry 1A through semi-structured one-on-one interviews. These interviewees were from several 
departments on campus, including the College of Chemistry, the Office of Media Services, the Berkeley 
Multimedia Research Center, the Instructional Technology Program, Information Systems and Technology and 
Student Services. We asked interviewees to identify: 

• teaching and non-teaching staff activities and time spent on each activity; 

• supplies and equipment (S & E) necessary for performing each activity (e.g., computers, software, 
chemical supplies); and 

• location where each activity was performed (e.g., lecture hall, office). 

During Year 1, we asked TAs to fill out weekly time logs, identifying their hours spent on in-class and out-of- 
class activities. We personally observed 16 and 68 randomly selected meetings of lab sessions in Years 1 and 2, 
respectively. In Y ear 2, TAs completed end-of-semester surveys reporting their time spent on typical weekly 
activities. We corroborated these self-reports with interviews, observations and our knowledge of TA activities. 

For each of the 50 activities, we collected cost data for the following major resource types: 

• “Staff’ resources, meaning the salaries and benefits of instructors, TAs and non-teaching staff; 

• “S & E” (supplies and equipment) resources, meaning the cost of using computer and other equipment, 
the cost of using software, and the cost of using supplies; and 

• “Facilities” resources, meaning the cost of using such facilities as lecture halls, labs and office space. 

As explained below, we measured costs using good proxies for the opportunity costs of resources used (Levin 
and McEwan, 2001). For example, to compute the cost of using a lecture hall, we used the cost of constructing 
additional lecture hall space, not the depreciation based on the historical cost of the building. 



4 Levin and McEwan (2001) use the term “ingredients,” which is essentially the same concept as activity-based costing. 

5 See Maher, Sommer, Acredolo and Matthews (2002). 
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lb. Staff Costs 

For each staff person — instructor, TA, and non-teaching staff — we computed a cost driver rate using the 
following formula: 

Cost driver rate = [staff person’s total salary and benefits] / [total hours employed] 

For example, if a TA was employed for 20 hours per week, we computed the rate as the TA’s weekly salary and 
benefits divided by 20 hours per week. 

We obtained salary and benefits information from the Chemistry Department administration, the university’s 
published schedule of pay rates, and staff interviews. 

lc. S & E Costs 

We obtained cost and usage information about consumable supplies, such as chemicals used in experiments, 
from the Chemistry Department. We amortized the costs of assets, such as equipment and software that have a 
multi-year life, using the following discounted cash flow formula (Levin and McEwan, 2001, pp. 65-70): 

e = C x [ r (l + r) n ]/[(l +r) n - 1 ], 

where 

e = estimated annual cost, 

C = initial cost of acquiring the asset, 
r = annual discount rate, and 
n = expected life of asset in years. 

This formula takes into account both the annual amortization and the opportunity cost of the funds used to 
acquire the asset. We used a five percent discount rate, which is at the high end of the range recommended by 
Levin and McEwan, but appropriate based on UC Berkeley’s experience in financing assets. 

We used three- to five-year life estimates for software and three- to eight-year lives for equipment. We divided 
the annual cost by three — for the three terms in which the course is offered (fall, spring, and summer) — to derive 
the costs per term. 

l d. Facility Costs 

For the facilities that Chemistry 1 A used, including lecture hall, lab space, and office space, we obtained the cost 
per square foot for new construction and the estimated life of facilities from the UC Berkeley Office of Planning 
and Analysis. These data indicated construction costs ranging from $150 per square foot for basic instructional 
facilities to $400 per square foot for the laboratory building, and an estimated facility life of 50 years. We 
measured each facility to obtain square footage data, and calculated the facility’s cost by multiplying the 
construction cost per square foot by the number of square feet. Chemistry 1 A used the lecture hall nine hours 
per week out of its practical capacity of 50 hours per week. Therefore, we assigned 1 8 percent (nine-fiftieths) of 
the lecture hall’s facility costs to Chemistry 1A. 

We amortized the cost of facilities used for Chemistry 1A using the amortization formula shown above and 
using the five percent discount rate discussed above. 

Many of the S & E and facilities’ resources were used for a single activity — lecture hall costs for the activity 
“delivering lectures” and chemical supplies costs for the activity “gathering chemicals for experiments,” for 
example. For those resources used for multiple activities, such as computers and PowerPoint software, we 
assigned costs to activities based on interviews with instructors, TAs, and non-teaching staff. 
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le. Infrastructure Costs 

For the purposes of this study, we included only costs that were directly relevant to course activities. We did not 
include the costs of utilities and maintenance of facilities nor the salaries of campus administrators, for example. 
Furthermore, we included only costs to the institution, not costs to the students. 

2. Results 

Tables 5 through 1 1 show the results of our cost analysis comparing the traditional and technology-enhanced 
courses for Year 1, while Tables 12 and 13 compare the costs of the technology-enhanced courses in Years 1 
and 2. The rows in each table show activities. We have clustered some of the 50 activities for ease of 
presentation. For example, we combined five activities for writing, proctoring and grading exams into one 
activity for presentation purposes. We show the detailed list of 50 activities and their related costs in 
Appendix A. 

The columns in Tables 5 through 1 1 show the Year 1 Traditional course costs, the Year 1 Technology-Enhanced 
course costs and the Savings provided by the technology-enhanced course, while the columns in Tables 12 and 
13 compare the costs of the technology-enhanced courses in Years 1 and 2. 

2a. Development/Revision Costs 

Development/Revision activities relate to the production or modification of course materials that will be 
reusable over multiple terms, such as producing the lecture slide presentations, programming computerized quiz 
questions, and rewriting the laboratory manual. The costs of development/revision activities, which appear in 
Table 5, were $68,731 higher for the technology-enhanced course, which is an incremental cost of $54.64 per 
student — less than the cost of a typical science textbook. These higher costs were mostly for preparing lecture 
graphics and overheads and designing and preparing the course website. Although not detailed in Table 5, 
nearly all of the additional costs were staff costs — $67,391 of the $68,731 total. 



Table 5: Development/Revision Costs 





Year 1 


Year 1 




Activity 


Traditional 


Technology-Enhanced 


Savings* 


Preparing lecture graphics and overheads 


$0 


$43,592 


($43,592) 


Creating and updating lab manual 


$873 


$873 


$0 


Revising discussion handbook 


$2,108 


$2,108 


$0 


Preparing and programming questions for on-line 
quizzes and assignments 


$0 


$3,075 


($3,075) 


Designing and preparing course website 


$0 


$22,064 


($22,064) 


TOTAL 


$2,981 


$71,713 


($68,731) 


Cost per Student (n = 1,258) 


$2.37 


$57.01 


($54.64) 



* traditional minus technology-enhanced 



While Table 5 presents development/revision costs that created capacity to use technology for many semesters, 
Tables 6 through 9 present our results of the costs of delivering the course each semester. 

2b. Delivery Costs — Lectures 

Table 6 presents the costs of activities related to lectures, such as the cost of delivering the lecture, preparing 
and conducting the chemistry demonstrations in lecture, and producing the lecture webcasts. In total, the 
technology-enhanced course saved $40,172 (savings of $31.93 per student) compared to the traditional course. 
The technology-enhanced course saved instructors a substantial amount of time in preparing for lectures by 
eliminating the need to put materials on chalkboards before class, which saved approximately 15 hours per 
week. In addition, by organizing the lecture materials in PowerPoint slides, the instructors saved time in 
collecting materials for class, organizing materials for the lecture, and practicing their lectures. This savings in 
lecture preparation — $47,259 (Table 6) — more than paid for the $43,592 investment in developing lecture 
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graphics and overheads (Table 5). This is a case where the development of technology paid for itself in the first 
year. 



Table 6: Costs of Delivering the Course, Lectures 





Year 1 


Year 1 






Traditional 


Technology-Enhanced 


Savings* 


Delivering the lecture, including demos and in-class video display 


$68,561 


$71,377 


($2,815) 


Encoding, streaming, archiving of lecture webcasts 


$0 


$2,837 


($2,837) 


Preparing for lecture (instructor) 


$77,391 


$30,132 


$47,259 


Setting up lecture browser presentation 


$0 


$1,435 


($1,435) 


TOTAL 


$145,952 


$105,780 


$40,172 


Cost per Student (n = 1,258) 


$116.02 


$84.09 


$31.93 



traditional minus technology-enhanced 



2c. Delivery Costs — Laboratories 

Table 7 presents the costs of activities related to laboratory instruction, such as teaching laboratory sections and 
providing chemicals for laboratory experiments in addition to the implementation of structured Discussion 
sessions. Although these activities accounted for nearly half of the total costs of offering Chemistry 1 A, we 
found virtually no cost difference between the traditional and technology-enhanced courses for laboratory 
activities. 

Although the costs do not appear to be different between the two versions of the course, in fact, these cost data 
fail to reveal important underlying phenomena for TAs’ activities. The TAs in the technology-enhanced version 
of the course (i.e., the treatment group) saved time in collecting and returning hard copy assignments and exams 
and in administering quizzes, time which they reallocated to working one-on-one with students, and to the 
Experiment and Discussion. 



Table 7: Costs of Delivering the Course, Laboratories 





Year 1 


Year 1 






Traditional 


Technology-Enhanced 


Savings* 


Supplying chemicals for experiments 


$70,976 


$70,976 


$0 


Attending TA meetings and lectures 


$138,773 


$138,773 


$0 


Teaching and administering lab sections 


$123,502 


$123,502 


$0 


Preparing for laboratory teaching (TAs) 


$54,265 


$54,265 


$0 


Recruiting, training, and monitoring teaching staff 


$36,577 


$33,797 


$2,780 


TOTAL 


$424,094 


$421,314 


$2,780 


Cost per Student (n = 1,258) 


$337.12 


$334.91 


$2.21 



* traditional minus technology-enhanced 



2d. Delivery Costs — Quizzes, Exams and Grading 

Table 8 presents the costs of activities related to performance assessment — quizzes, exams, and grading — which 
are performed outside of lecture and laboratory sections, and may cover material from either lecture or 
laboratory or both. The development of on-line quizzes and assignments resulted in the major savings in the 
technology-enhanced course. Comparing these savings to the costs of developing on-line quizzes and 
assignments in Table 5 plus the delivery costs of on-line quizzes and assignments in Table 8, we see that on-line 
quizzes and assignments provided a substantial net benefit, computed as follows: 
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Cost of technology enhancement — on-line quizzes and assignments: 



Table 5, development costs $3,075 

Table 8, delivery costs 5,706 

Total costs of technology enhancements $8,781 

Cost savings from on-line quizzes and assignments, Table 8 $ 23,063 

Net benefit from on-line quizzes and assignments $ 14,282 



Our results parallel those at the University of Wisconsin-Madison, which also reported a savings in TA time 
from putting homework and quizzes on-line (see for example, the Pew Grant Program in Course Redesign 
website). 6 



Table 8: Costs of Delivering the Course: Quizzes, Exams, and Grading 





Year 1 
Traditional 


Year 1 

Technology-Enhanced 


Savings* 


Grading homework quizzes, check-in quizzes, 
and pre-lab exercises 


$32,559 


$9,496 


$23,063 


Grading lab write-ups 


$88,633 


$88,633 


$0 


Creating quizzes (traditional version) 


$462 


$0 


$462 


Grading (instructor) 


$5,560 


$5,560 


$0 


Setting-up/maintaining on-line assignments/quizzes 


$0 


$5,706 


($5,706) 


Writing, proctoring, and grading exams 


$85,910 


$84,428 


$1,483 


TOTAL 


$213,125 


$193,823 


$19,302 


Cost per Student (n = 1,258) 


$169.49 


$154.07 


$15.34 



* traditional minus technology-enhanced 



2e. Delivery Costs — General 

Some activities were not directly associated with any of the above three categories, but instead provided the 
basic infrastructure to support instruction in each of these areas. These general course-related activities included 
enrolling students, maintaining the student database, course-level planning and administration, and design and 
maintenance of the general course website. Table 9 shows that the cost of these activities was higher for the 
technology-enhanced course. Both maintaining and hosting the website and student enrollment and related 
activities cost more for the technology-enhanced course. These higher costs were offset somewhat by cost 
savings in instructor office hours for the technology-enhanced course. 



Table 9: Costs of Delivering the Course, General 





Year 1 


Year 1 






Traditional 


Technology-Enhanced 


Savings* 


Course-level administration and scheduling 


$22,019 


$22,019 


$0 


Office hours (instructor) 


$27,465 


$19,741 


$7,723 


Maintaining and hosting website 


$0 


$12,257 


($12,257) 


Tutoring and other out-of-class student support 


$11,400 


$11,400 


$0 


Out-of-class teaching activities 


$103,542 


$103,656 


($114) 


Student enrollment, account, and database management 


$3,753 


$9,493 


($5,740) 


Updating, editing, and publishing the syllabus 


$3,074 


$2,703 


$371 


TOTAL 


$171,252 


$181,269 


($10,017) 


Cost per Student (n = 1,258) 


$136.13 


$144.09 


($7.96) 



* traditional minus technology-enhanced 



6 Unlike the University of Wisconsin, our cost analysis was more detailed and covered the entire cost of the course, including such things 
as chemical supplies. 
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2f. Summary of All Course Costs 

Table 10 presents the “bottom line” of cost comparison. Overall, the technology-enhanced course cost $16,494 
more ($13.11 per student), which was less than two percent of total costs. All things considered, the 
technology-enhanced course’s additional development costs of $68,731 were mostly offset by the first year 
savings in delivery: costs amounting to $52, 237. 7 The single largest cost savings, $47,259, came in lecture 
preparation, as noted earlier in this section. Note that we did not amortize development costs. It is reasonable to 
expect that the development costs for the Year 1 technology-enhanced course will generate savings in future 
year delivery costs. 



Table 10: Summary of All Course Costs 





Year 1 


Year 1 






Traditional 


Technology-Enhanced 


Savings* 


Dcvelopment/Revision 


$2,981 


$71,713 


($68,731) 


Delivery: 








Lectures 


$145,952 


$105,780 


$40,172 


Laboratories 


$424,094 


$421,314 


$2,780 


Quizzes, Exams, and Grading 


$213,125 


$193,823 


$19,302 


General 


$171,252 


$181,269 


($10,017) 


TOTAL 


$957,405 


$973,899 


($16,494) 


Cost per Student (n = 1,258) 


$761.05 


$774.16 


($13.11) 



traditional minus technology-enhanced 



2g. Breakdown by Source of Cost 

Table 1 1 shows that more than 90 percent of all costs are staff salaries and benefits, from which we infer that 
using technology to create cost savings should focus on personnel costs. In particular, if the university seeks to 
manage costs in the face of rising enrollments, then the university should find ways to reduce TA activities (and 
therefore, TA costs). 



Table 11: Breakdown by Source of Cost 





Year 1 
Traditional 




Year 1 

Technology-Enhanced 


Savings* 


Staff: 

Instructors 


$152,940 


16% 


$132,065 


14% 


$20,875 


TAs 


$619,887 


65% 


$598,366 


61% 


$21,521 


Non-Teaching Staff 


$97,154 


10% 


$149,210 


15% 


($52,057) 


Supplies & Equipment** 


$48,767 


5% 


$54,907 


6% 


($6,140) 


Facilities 


$38,657 


4% 


$39,350 


4% 


($694) 


TOTAL 


$957,405 


100% 


$973,899 


100% 


($16,494) 



* traditional minus technology-enhanced 
** including computer software and hardware 



For Chemistry 1A, technology enhancements required additional resources from non-teaching staff, for example 
to design and prepare the course website, resulting in additional non-teaching staff costs of $52,057. Note the 
cost savings for instructors and TAs in Table 11. Would the university reduce payments for instruction or 
reduce the number of TAs hired? The answer to that question is beyond the scope of this study, but the cost 
savings reported in Table 1 1 indicates freed up time for instructors and TAs. 



7 Total delivery cost savings amounted to $52,237 in Year 1 ($954,423 - $902,186; See Table 14). 
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2h. Comparison of Year 1 and Year 2 Development/Revision Costs 

Table 12 compares the development/revision costs for the technology-enhanced course in Years 1 and 2. For 
this analysis, we used the same salary, benefits, and other costs of activities in Y ear 2 as in Y ear 1 to make the 
years as comparable as possible. 

Table 12: Comparison of Year 1 and Year 2 Development/Revision Costs 
in Technology-Enhanced Courses 



Activity 


Year 1 

Technology-Enhanced 


Year 2 

Technology-Enhanced 


Cost Savings 
in Year 2* 


Preparing lecture graphics and overheads 


$43,592 


$10,615 


$32,977 


Creating and updating lab manual 


$873 


$378 


$495 


Revising discussion handbook 


$2,108 


$2,193 


($85) 


Preparing/programming on-line quizzes/assignments 


$3,075 


$1,228 


$1,847 


Designing and preparing course website 


$22,064 


$9,105 


12,959 


TOTAL 


$71,713 


$23,519 


$48,194 


Cost per Student (n = 1,258 Year 1, n = 1,202 Year 2) 


$57.01 


$19.57 


$37.44 



* Y ear 1 minus Y ear 2 



Comparing Years 1 and 2 shows a substantial decrease in development/revision costs in Year 2 — $48,194, or 
about two-thirds of the Year 1 development/revision costs. After instructors and non-teaching staff put in the 
time for such technology enhancements as developing the lecture slides and designing the course website for 
Year 1, their time requirements were reduced in Year 2. Does Year 2 represent a steady-state level of activities? 
If it does, then the technology-enhanced course would likely have lower costs than the traditional course, as 
explored further in the next section. 

Note that the two years are not perfectly comparable because of the different cost driver volumes. Year 2 had 
56 fewer students and three fewer TAs than Year 1, for example. Nevertheless, these differences between years 
are not large enough to explain the cost differences in Table 12. 

2i. Comparison of Year 1 and Year 2 Course Delivery Costs 

Table 13 compares the costs of delivering the technology-enhanced course in Years 1 and 2. As we did for the 
development/revision costs, we used the same salary, benefits, and other costs of activities in Year 2 as in 
Year 1 to make the years as comparable as possible. As Table 13 shows, costs were about five percent lower in 
Year 2, which, considering three fewer lab sections and 56 fewer students in Year 2, implies little additional cost 
savings from technology enhancement in delivering the course. As noted above, the development/revision costs 
were substantially lower in Year 2 than in the Year 1 technology-enhanced course. 

Table 13: Comparison of Year 1 and Year 2 Course Delivery Costs 
in Technology-Enhanced Courses 



Activity 


Year 1 

Technology-Enhanced 


Year 2 

Technology-Enhanced 


Cost Savings 
in Year 2* 


Lectures 


$105,780 


$95,727 


$10,053 


Laboratories 


$421,314 


$411,091 


$10,223 


Quizzes, Exams and Grading 


$193,823 


$138,246 


$55,577 


General 


$181,269 


$174,834 


$6,435 


TOTAL 


$902,186 


$819,898 




Cost per Student (n = 1,258 Year 1, n = 1,202 Year 2) 


$717.16 


$682.11 


$35.05 | 



Y ear 1 minus Y ear 2 



Table 14 compares the total course costs in the traditional course, Year 1 of the technology-enhanced course, 
and Year 2 of the technology-enhanced course. Comparing the Year 2 cost per student of $701.68 to the cost 
per student of $761.05 for the traditional course in Year 1 reveals nearly $60 per student cost savings. 
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We caution the reader that there may be some problems with this comparison because of different activity levels 
(e.g., number of students, number of TAs). In addition, Year 2 of the study saw a reorganization of the campus 
technology support units and increases in the associated costs, the hiring of an additional full-time course staff 
person (lab manager), and the streamlining of TA grading procedures. Nevertheless, there appears to be a real 
cost savings in technology enhancements, even without amortizing the time spent developing them. 

Table 14: Total Course Costs I 





Year 1 
Traditional 


Year 1 
Technology- 
Enhanced 


Year 2 
Technology- 
Enhanced 


Year 2 
Savings* 


Development/revision 


$2,981 


$71,713 


$23,519 


($20,538) 


Delivery 


$954,423 


$902,186 


$819,898 


$134,525 


TOTAL 


$957,405 


$973,899 


$843,417 


$113,987 


Cost per Student 

(Yl, n = 1,258; Y2, n=l,202) 


$761.05 


$774.16 


$701.68 


$59.37 



* Y1 traditional minus Y2 technology-enhanced 
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B. Teaching Staff Time and Attitudes 

1. Data Collection 

One focus of this study is the question of redistribution of staff time; therefore, we used a variety of methods to 
collect data on teaching staff (instructors and TAs) time and attitudes over the two-year study. These methods 
included interviews, student and TA surveys, time logs, observations, and focus groups. Because of the sheer 
numbers and expense of TAs, a large part of our data collection focused on their time, especially in labs. We 
were particularly interested in whether the technology enhancements changed the way TAs used time in the 
labs. Assessing TA time, however, was fraught with difficulty because of inconsistency in self reports of time. 
Therefore we relied on triangulation of many data sets (time logs, observations, surveys) as detailed below. 

la. Teaching Assistants 

Table 15 summarizes participation of TAs in the research project. During Year 1, TAs completed weekly time 
logs and a survey at the end of the semester. We also conducted observations of randomly selected lab sections 
and focus groups. In Year 2, due to TA inconsistency and some resistance to time logs in the previous year, we 
decided to substitute direct observations as a better measure of time spent in labs, and conducted partial 
observations (see “Direct Observation” below) on a larger set of lab sections. It should be noted that each year 
of Chemistry 1A involves a new cohort of TAs, most of whom are first-year graduate students. 



Table 15: TA Participation in the Research Project 





Year 1 | 


j Year 2 1 




Total TAs 

% 


N 


Control Group 

% N 


Treatment Group 

% N 


Total TAs 

% 


N 


Total number of TAs* 


100% 


45 


69% 


34 


22% 


11 


100% 


41 


Consented to general project 


82% 


37 


79% 


27 


91% 


10 






Responded to Pre-survey 


n/a 




n/a 




n/a 




g ii/rV- 




Responded to Mid-survey 


n/a 




n/a 




n/a 




76% 


31 


Responded to Post-survey 


80% 


36 


n/a 




n/a 




68% 


28 


Participated in Brief Interviews 


n/a 




n/a 




n/a 




46% 


19 


Participated in Focus Groups 


16% 


8 


12% 


4 


36% 


4 


n/a 




Turned in time logs 


82% 


37 


79% 


26 


100% 


11 


n/a 




Observed in lab 


8% 


4 


6% 


2 


18% 


2 


98% 


40 




Total Lab Section 
Meetings 
% N 


Control Group 

% N 


Treatment Group 

% N 


Total Lab Section 
Meetings 
% N 


Total lab section meetings 


100% 


630 


76% 


476 


24% 


154 


100% 


588 


Full observations 


2.5% 


16 


1.7% 


8 


5.2% 


8 


2.0% 


12 


Partial observations 


n/a 




n/a 




n/a 




9.5% 


56 


Total observations 


2.5% 


16 


1.7% 


8 


5.2% 


8 


11.6% 


68 



* Instructional TAs only. Each year there was also an additional e-TA and multiple Head TAs (3 in Year 1 ; 2 in Year 2). 



TA Time Logs. During Year 1, TAs were asked to fill out time logs on a weekly basis. We provided forms for 
the TAs with descriptions of activities related to their weekly Chemistry 1A schedule. We investigated a subset 
of TA self-reports by comparing time logs from TAs in observed sections in order to check for reliability. The 
difference between reported and observed time in lab during Y ear 1 shows a nine-minute overestimation on the 
part of the TAs on average, which falls well within the standard deviation range (Table 16). 

Table 16: Time in Lab, TA Report vs. Direct Observation, Year 1, Reported in hh:min 





Time Logs 


Lab Observations 




N 


Mean ± SD 


Mean ± SD 


Difference 


14 


3:30 ±0:35 


3:21 ±0:31 


0:09 
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Direct Observation: Full and Partial Observations. In Year 1, four lab sections (two sections in the control 
group and two in the experimental group) were selected for observation after students were enrolled and TAs 
section assignments were made. Each section was observed four times for a total of 16 lull lab observations. 

In Y ear 2, three lab sections were randomly selected for four observations each during the semester for a total of 
12 full lab observations. Additional lab sections in Year 2 were observed throughout the semester by a scanning 
process in which observers rotated throughout six lab sections during the first and last hours of the scheduled 
time period (i.e., either 8:00a.m. to noon, or 1:00p.m. to 5:00p.m.). A total of 68 laboratory section meetings 
were observed. 8 

TA Surveys. The Y ear 1 survey was responded to anonymously by TAs at the end of the semester. TAs in 
Y ear 2 were polled three times during the semester, with their identities recorded. The surveys included 
questions covering TA experience with educational technologies used in Chemistry 1A, preference for 
educational technologies in general, and technical/computer skills; and they also gave TAs the opportunity to 
offer suggestions for improving the course and course technologies. 

TA Focus Groups and Interviews. In Year 1, a small number of TAs participated in focus groups conducted by 
the research team. We did not conduct focus groups in Year 2. In Year 2, brief ad hoc TA interviews were 
conducted at the conclusion of the partial observations. 

2. Results 

2a. TA Profile 

Our observations, interviews, and TA surveys noted that there was a great range of variation in TA background, 
teaching experience and knowledge of educational technology tools. 

Teaching Experience. Of the Year 2 survey respondents, 36 percent of TAs reported that they had no teaching 
experience. Sixty-four percent reported some teaching experience of either a semester or more. Of those TAs 
with teaching experience, the great majority (86%) claimed that they gained teaching experience while 
undergraduates. 9 Less than one-third (29%) of experienced TAs had graduate-level teaching experience. Of 
those TAs with teaching experience, 76 percent said they taught chemistry. Nineteen percent indicated some 
other science. 

Technology Access and Ability. Overall, TAs reported moderate to high levels of technology access and ability. 
Most Year 2 survey respondents had previous experience with educational technologies such as course websites, 
lecture slides, on-line grades, and on-line quizzes. Fewer TAs had experience with either webcasts or on-line 
office hours. In short informal interviews, a large number of TAs (68%) reported using email or a homemade 
listserv to communicate regularly with students in their sections but only a few ( 1 0%) used the chat rooms 
provided. 

Attitudes about Technology and Teaching. All TAs (100%) responding to the Year 1 post-survey indicated 
that grading and tasks related to grading were a burdensome process. 10 More than one-third (39%) of TAs 
specifically noted the lab write-ups as time-consuming. One quarter (25%) of TAs referred to entering grades as 
being an unnecessary task. TAs in the control group also reported spending more time on grading outside of 
class than TAs in the treatment group. 



s For a few observations (5 of 68, 7%), observers were unable to observe the exact end-time of the Experiment or lab section meeting 
because it ended early. In these cases, we used the most conservative estimate, given our observation data. 

9 Most (86%, N=18) of those TAs who had experience indicated that they gained experience while they were undergraduates. It is 
possible that TA respondents may have perceived one-on-one tutoring experience as equitable to classroom teaching experience. 
Therefore, the actual number of TAs with classroom teaching experience may be lower than reported. (Note that undergraduates make 
up approximately 14% of Chemistry 1 A TAs during fall semesters and 100% in summer courses). 

10 In Year 1, TAs were required to keep a handwritten grade book in addition to multiple digital databases. They could not upload this 
information into a central database; rather they had to enter all grades multiple times from a single computer. This process was 
simplified in Year 2 with the implementation of a centralized on-line grades database that TAs could access anytime, anywhere. Though 
TAs must still maintain a handwritten grade book for departmental reasons, on-line assigmnents are uploaded automatically. 
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Of note is that more than 80 percent of Year 1 TAs surveyed reported that they would be willing to migrate 
in-class administrative tasks (e.g., quizzes) to an on-line environment. This, then, is an area where technology 
could potentially improve the TA experience. In Year 2, the majority of TAs felt that technology was a 
timesaver as well, though their opinions improved and became more widespread as they became more familiar 
with Chemistry 1A technologies. For example, the overwhelming majority (90%) of Year 2 TAs reported in the 
pre-survey that they felt that computer-based technology was a useful resource or teaching aid. More than half 
(58%) of TAs cited time-savings in general as the primary reason for this opinion. As the semester progressed 
and TAs’ experience with technology and teaching increased, their opinions of technology's role in the course 
also improved. By the mid-semester survey, most TAs (75%) claimed that technologies allowed them to save 
time in general and the vast majority (82%) agreed with this statement in the post-survey. 

TA optimism about whether technology could enable them to reallocate time in the classroom increased 
throughout the semester. In the Year 2 post-survey, more than half (54%) of TAs believed that technologies 
allowed them to spend more time with students compared to only one-third (33%) of respondents in the pre- 
survey. Although TAs perceived the technology as useful for saving time, they apparently did not perceive it as 
a rationale for shortening labs. 

TA opinions about the utility of technology to improve student learning also changed. In the pre-survey, 

27 percent of TAs indicated that they believed that on-line activities improved the student learning experience 
compared to 18 percent of TAs in the post-survey. This perception mirrors that of students who see technology 
in the classroom as a useful tool for time management rather than a replacement for the in-person learning 
experience. 

2b. TA Time in Lab Section Meetings 

Partitioning of Time in Labs 

Table 17 shows the breakdown of scheduled time in lab section. Administrative responsibilities, such as 
administering the homework quizzes and collecting or returning pre-laboratory assignments or exams, were 
expected to take about 15 minutes and were scheduled at the beginning of the lab section for the control group. 
In the treatment group, where some of the work was transferred on-line, TAs were not required to administer a 
homework quiz or to collect or return pre-laboratory assignments because students completed them on-line. As 
a result, TAs in the treatment group often did not use this time for administrative purposes and instead began 
with the required Discussion. 

Table 17. Scheduled Time in Lab Sections, Year 1, Reported in hh:min 



Section Type 


Homework Quiz 


Administration 


Discussion 


Experiment 


Control Group 


0:05 


0:10 


0:45 


3:00 


Treatment Group 




Partial 


0:45 


3:00 



Self reports and observations indicate that actual time spent in lab section meetings varied by week, TA, and 
individual student. Overall, TAs spent as much time in lab section meetings as necessary to meet students' 
needs. Observations in Year 1 revealed that TAs in treatment groups often reallocated saved time in the 
classroom to spend more time on activities related to student learning (e.g., treatment group TAs spent more 
time working with students individually, and more time on the Experiment and Discussion). TAs in the 
treatment group also reported spending less time on grading activities, which appears to be related to students’ 
on-line quiz-taking patterns. Observations indicate that the ability to manage time effectively varied greatly 
among individual TAs, which can most likely be attributed to experience, individual dispositions, and personal 
time -pressures. It is worth noting that student survey data indicate that students in the treatment groups were less 
likely to feel “rushed” in labs. Details of the two-year analysis are presented below. 
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Self Reports of Overall TA Time, Year 1 and Year 2 

In the first year of the study, TAs reported in time logs that they spent an average of 1 8 hours each week in 
activities related to Chemistry 1A. TAs in the control group spent an average of 18 hours and 23 minutes per 
week compared to TAs in the treatment group, who spent an average of 17 hours and 13 minutes per week in 
activities related to their duties (Table 18). 

TAs in Y ear 2 reported spending an average of 1 8 hours and 43 minutes per week on activities related to 
teaching Chemistry 1A (Table 18). TAs in Year 2 were only asked to report time on task once at the end of the 
semester whereas their Year 1 counterparts reported weekly in time logs. 



Table 18: TA Weekly Time, Self-Reported in hh:min, Mean ± SD 







Year 1 

TA Time Logs 




Year 2 
TA Survey 




All 

N=467 

N=3 1/wk average 


Control Group 

N=347 

N=23/wk average 


Treatment Group 

N=120 

N=8/wk average 


All 

N=28 


Required 

lecture, office hours, TA meeting 


5:39 ± 1:29 


5:43 ± 1:24 


5:28 ± 1:42 


6:19 ± 1:24 


Lab Section 


3:28 ± 1:16 


3:30 ±1:18 


3:22 ±1:13 


4:12 ±0:48 


Grading 

includes exam proctoring 


5:37 ±4:36 


5:45 ±4:43 


5:11 ±4:14 


4:20 ±2:04 


Preparation 


2:04 ± 1:40 


2:02 ± 1:44 


2:08 ± 1:29 


2:04 ± 1:30 


Miscellaneous 


1:16 ±2:28 


1:21 ±2:49 


1:01 ±0:54 


1:41 ± 1:16 


Total time spent on Chemistry 1A 


18:05 ±6:09 


18:23 ±6:22 


17:13 ±5:27 


18:43 ±4:47 



Time logs from TAs in the treatment group also suggest a savings of time in three areas outside of lab: grading 
(34 minutes on average), required activities 11 (15 minutes on average), and miscellaneous activities (20 minutes 
on average). TAs in the treatment group, however, reported spending more time preparing for lab section 
meetings (16 minutes) and conducting the experiment (7 minutes). In weeks when there were homework 
quizzes implemented in the control group, TAs in the treatment group were observed spending more time 
assisting students in conducting the experiments. 

Year 1 TA Time: Observations, Control vs. Treatment Groups 

Observed Time in Labs, Year 1. Table 19 summarizes the results of the data we collected on the time TAs 
spent in lab section meetings. Year 1 observations revealed that TAs and students in the treatment group 
appeared to spend slightly more time on the Discussion and Experiment (four minutes on average per activity) 
than those in the control group. In summary, observations and surveys indicate that TAs spent less time on 
administrative tasks in the classroom, such as administering quizzes, in Year 1 as a result of the technology. 



1 1 TAs are required to attend three lectures and the TA meeting, and conduct office hours each week. As requirements were not different 
for the TAs in either group, we suspect that this timesavings is an anomaly due to the specific group of TAs sampled. 
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Table 19: Year 1 and Year 2 TA Time, Observed, Reported in hh:min 



Year 1 

Full Observations 


Year 2 

Full and Partial 
Observations 




All Lab Sections 
Mean ± SD 


N 


Control Group 
Mean ± SD 


N 


Treatment Group 
Mean ± SD N 


All Lab Sections 
Mean ± SD* N 


Total TA Time in Lab 


3:12 ±0:38 


16 


3:13 ± 0:30 


8 


3: 12 ±0:47 


8 


3:32 ±0:34 68 


Total Class Time 1 


2:44 ±0:39 


16 


2:43 ±0:37 


8 


2:45 ±0:43 


8 


2:37 ±0:42 43 


Classroom Activities: 
















Homework Quiz 2 


0:15 ±0:03 


4 


0:15 ±0:03 


4 


n/a 




n/a 


Discussion 2 


0:46 ±0:12 


12 


0:44 ±0:16 


6 


0:48 ±0:08 


6 


0:48 ±0:16 23 


Experiment 2 1 


1:38 ±0:25 


16 


1:36 ± 0:18 


8 


1:40 ±0:33 


8 


2:15 ±0:46 63 


TA Behaviors: 
















Administration 


0:15 ±0:07 


8 


0:21 ±0:05 


4 


0:09 ±0:04 


4 


n/a 


TA instructional time 


0:58 ±0:38 


16 


1:02 ±0:41 


8 


0:54 ±0:33 


8 


n/a 


One-on-one student interaction 


1:22 ±0:41 


16 


1:09 ±0:20 


8 


1:36 ±0:53 


8 


n/a 



* Most conservative estimate. 

1 In Year 1, Class Time is calculated as the time from TA arrival to the time the last student completed experiment whereas in Year 2, it is 
calculated as the time from the first official class activity to the time the last student completed the experiment. 

2 Individual activities in the lab section meeting do not necessarily represent a breakdown of “class time” for either students or TAs because of 
additional time spent in other types of activities. 

' SD in experiment time shows difference between experiments more than difference between TAs. 



In addition to recording duration of class time devoted to the Discussion and Experiment, we were also able to 
observe the quality and duration of interactions among TAs and students in Y ear 1 . These observations revealed 
that TAs in control and treatment groups allocated time differently in the classroom. TAs in the treatment group 
spent more time in class interacting one-on-one with students. In fact, they appeared to spend more than half of 
the lab section meeting (50%, 1 hour and 36 minutes on average) working individually with students while TAs 
in the control group spent only slightly more than one-third of their time (36%, 1 hour and 9 minutes on 
average) in similar activities. 

Observations in Y ear 1 also showed that TAs in the treatment group saved an average of 20 minutes in 
instructional and administrative time. The savings came from less time spent: lecturing or at the chalkboard, 
conducting the check-in question or the homework quiz, and performing other administrative functions. 

Year 2 TA Time: Observations 



Based on Year 1 lab observations, we concluded that labs could be shortened from four to three hours (with 
some changes to the curriculum, such as limiting the unpopular Discussion). The Department agreed to explore 
the idea of a shorter lab section meeting in Year 2 for all lab sections. This was accomplished with a minimum 
of disruption by keeping the four-hour time slot, but designating the first hour as optional for students. Interim 
surveys administered three weeks into the semester during Y ear 2 indicate that approximately half of TAs felt 
time constraints with a three-hour lab section meeting. Some TAs reported that the reduced time limit for lab 
section meetings negatively affected their ability to answer questions and explain concepts to students. As a 
result, course staff gave TAs the option to allocate time at their own discretion and many TAs opted to return to 
the scheduled four-hour time. 

Our Year 2 observations of 68 lab section meetings, however, indicate that most lab section meetings were 
under four hours, with many taking less than three hours (Table 19). Depending on the assigned Experiment for 
that day, the total time spent in necessary class activities ranged anywhere from 32 minutes to three hours and 
5 1 minutes. The Experiment made up the bulk of the lab period. 
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2c. Year 1 Student Time: Self Reports 

To expand on our observations of TA time in labs, we collected survey data on student time both in and out of 
lab section meetings in Year 1. A complete description of the student surveys can be found in Section D 
(Student Attitudes About the Course). 

Overall Student Time 



According to self-reported data collected in the on-line post-survey, students in Year 1 reported spending an 
average of 14.8 hours per week on activities related to Chemistry 1A overall. Students reported spending an 
average 7.3 hours per week in activities related to lecture. There were only slight differences between control 
and treatment groups for the mean values. Students reported in the on-line post-survey in Year 1 that they spent 
an average of 7.6 hours per week on activities related to lab. There was no real difference between control and 
treatment group self reports regarding the total number of hours spent on activities related to lab. 

Student Time in Labs 



Respondents to the Year 1 on-line post-survey generally felt that there was enough class time during their lab 
sections to ask questions about lecture (58% responding “most of the time” or “always”). Of those respondents, 
students in control group (57% responding “most of the time” or “always”) were somewhat less likely to feel 
this way than were students in treatment group (60% responding “most of the time” or “always”). 

Although most (59%) of the on-line post-survey respondents said that they “never” felt that there was 
insufficient time during their lab section to complete the experiment, 41 percent did state that at least “some of 
time” there was not enough time to complete their experiment. This difference becomes more apparent when 
viewed by section type: many more students in the control group (50%) said that at least “some of the time” 
they had problems completing in-section tasks than did students in the treatment group (22%). This difference 
is statistically significant ( yf= 14.8, p = 0.0001). 

In looking at student time observed in labs in both control and treatment groups in Year 1, it appears that 
students enrolled in sections in the treatment group did not spend as much time on administrative tasks. On 
average, administrative tasks occupied 21 minutes of class time in control group sections and only 9 minutes of 
class time in treatment groups. However, during these observed weeks, students in the treatment group spent 
approximately the same amount of time in section as students in the control group, indicating that students and 
TAs reallocate time from administrative duties to other activities. 

2d. Instructor Time 

Staff Interviews. Section A (Measuring Cost Effectiveness) reports on much of the data we collected regarding 
teaching and non-teaching staff time. The relevant findings related to instructor time in particular are 
summarized here. A total of nineteen interviews with teaching and non-teaching staff were recorded and 
transcribed. Interviews were conducted with faculty and staff from the College of Chemistry as well as from 
campus units involved in technology support and implementation. We also interviewed others associated with 
the College of Chemistry, such as the dean and instructors of Chemistry 1A during other semesters. 

Instructor Time and Attitudes 



Before discussing results, it must be noted that the introductory chemistry course at UC Berkeley is taught by 
tenure-track faculty who are active researchers. This is in contrast to many Research One and other institutions 
where a dedicated teaching staff of lecturers or adjuncts has the primary responsibility for teaching large 
introductory courses. At UC Berkeley, each faculty member has a distinct philosophy, strong preferences, and 
considerable flexibility in how to teach the class. 

The lead instructor for Chemistry 1 A estimated an overall time-savings of 53 percent per week due to 
technology enhancements to the course in Year 2 as compared to the traditional version of the course. Rather 
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than spend less time on the course overall, the instructor reported that he spends the saved time on other 
activities related to improving instruction and course development. 

In particular, our data show that the lead instructor spends considerably less time (35%) preparing for the 
technology-enhanced class as a result of the lecture slides. The bulk of these savings can be attributed to the 
eliminated need for extensive chalkboard preparation prior to lectures. (Although the lead instructor saved 
considerable time on chalkboard preparation, three times as much time was spent in Year 1 by the second 
instructor and an additional student assistant on the initial development of the digital lecture slides.) While 
some time is still spent in Year 2 on revising and preparing lecture slides, much less time is spent (87% less 
overall) in Year 2 than in Year 1. Savings were also noted by both instructors in lecture preparation and 
organization. Both instructors noted that time spent consulting about revisions to each lecture’s total scope and 
sequence was facilitated by the electronic format of the slides. Collective time spent on these preparatory 
activities was reduced by an additional 1 0 percent. 

The lead instructor spent less time answering routine student questions about course administration in office 
hours or by email. He reported that student contact time was reduced by more than half in the technology- 
enhanced course, which is an additional weekly time-savings of six percent. Student survey responses confirm 
this. We found that 69 percent of Year 1 students and 62 percent of Year 2 students stated that they have used 
e-mail to ask questions of instructors instead of going to office hours in person either always, most, or some of 
the time. Additionally, thirteen and eight percent, in Year 1 and 2 respectively, reported that they consulted 
webcasts instead of seeking assistance from a TA or Instructor. 

Information provided by other chemistry instructors suggested that the distribution of development and delivery 
time varies from instructor to instructor because of personal teaching styles and philosophies. The role of 
educational technology in particular is not viewed in a uniform way by either instructors or by administrators. 
For example, sophisticated electronic chemistry modules developed some years ago by another faculty member 
in the department are not used by many other chemistry instructors. Additionally, some of the on-line resources 
(quizzes) that are the subject of this report were adopted in a subsequent semester of Chemistry 1A, but were 
abandoned the second year because of significant implementation problems and lack of support staff necessary 
to ensure success. So far, the lecture slides have not been adopted by other instructors. We should note that 
administrators were generally unaware of the ways in which technologies might be used to reconfigure space 
and time. Instead, most view technology as having the potential to improve course pedagogy (e.g., tutorials, 
simulations, data sets); issues of cost-savings and/or increasing convenience for faculty or students are rarely 
mentioned as reasons for investing in educational technology. 
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C. Student Background, Performance, and Retention 

1. Data Collection 

la. Background Information 

In both years, demographic data, such as age, gender, race/ethnicity, and socioeconomic status, as well as 
descriptive measures from college and high school, such as GPA, units completed, college major, and SAT 
scores, were collected from individual consenting students’ university records. Additionally, the same aggregate 
data were collected for all students who completed the course. Table 20 provides a profile of students taking the 
course. 



Table 20: Summary of Student Demographics and Background Data, Year 1 and Year 2 





Year 1 (N=l,331*) 


Year 2 (N=l,135**) 




Frequency (%) Mean ± SD 


Frequency (%) Mean ± SD 


Demographic Measures 






Gender: Male 


591 (44%) 


506 (45%) 


Female 


740 (56%) 


623 (55%) 


Ethnicity: White 


363 (27%) 


295 (26%) 


Black 


36 (3%) 


19 (2%) 


Hispanic 


83 (6%) 


81 (7%) 


Asian/Pacific islander 


687 (52%) 


605 (54%) 


Other 


25 (2%) 


20 (2%) 


Age 


18.5 ± 1.6 


18.8 ±0.63 


First Learned Language: English only 


593 (45%) 


476 (42%) 


Another language only 


322 (25%) 


269 (24%) 


English & another language 


398 (30%) 


379 (34%) 


Socioeconomic status 






Father’s education: No high school 


77 (6%) 


66 (6%) 


Some high school 


45 (4%) 


35 (3%) 


High school grad 


103 (8%) 


92 (9%) 


Some college 


115(9%) 


92 (9%) 


2-year college grad 


43 (3%) 


43 (4%) 


4-year college grad 


312(25%) 


283 (27%) 


Post-grad 


538 (44%) 


448 (42%) 


Mother’s education: No high school 


104 (8%) 


83 (8%) 


Some high school 


40 (3%) 


37 (3%) 


High school grad 


159(13%) 


130 (12%) 


Some college 


155(12%) 


110(10%) 


2-year college grad 


96 (8%) 


84 (8%) 


4-year college grad 


406 (32%) 


369 (34%) 


Post-grad 


307 (24%) 


266 (25%) 


Parental income 


$81,748 ± 


$90,024 ± 




$82,412 


$83,202 


Educational Background 






SAT I : Verbal score 


638 ± 90 


632 ±91 


Math score 


686 ± 79 


686 ± 78 


Total score 


1324 ± 145 


1318 ± 147 


SAT II (Achievement Test) 


2002 ± 224 


2006 ± 226 


High school GPA 


4.26 ±0.33 


4.27 ±0.28 


First semester of college 


1,114(87%) 


966 (86%) 



* Y ear 1 : includes students enrolled in the course at any point during the semester. 
** Year 2: includes students who completed the course. 
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lb. Student Performance 

We examined the impact of the technology enhancements on three aspects of student performance and attitudes: 
1) student learning as measured by exam performance and a carry-forward experiment, 2) course retention rates, 
and 3) student attitudes regarding the course. The discussion below covers student learning and retention in the 
Year 1 control and treatment groups. A similar comparison was not possible in Year 2. Student attitudes in 
Year 1 and Year 2 are reported in Section D (Student Attitudes About the Course). We also looked at the 
relationship among certain types of on-line behavior, performance, and attitudes in both years. Those results are 
in Section E ( Student Use of Technology Enhancements). 

Students’ Grades and Coursework. Course grades were determined by a number of measures, including final 
exam, mid-term, and final course grades. We measured student performance through scores on course exams 
(midterms and finals), and other coursework. All students completed the same lab assignments and took the 
same exam s . 

Carry-Forward. In addition to the direct comparisons described above, during Year 1 we worked with faculty 
in the succeeding course (“Chemical Structure and Reactivity”) to measure student understanding of core 
concepts from both classes. This method, a “carry-forward” experiment (Seymour, 1999), can help identify how 
the on-line tools help students learn and retain knowledge. Data collection for the carry-forward experiment 
started in the fall semester of Year 1 with the follow up data collected in the spring semester of that same 
academic year. A total of 272 students completed the subsequent chemistry course and consented to release 
their course information. 

lc. Course Retention Data 

Data regarding course completion were collected at the end of the semester and compared across the control and 
treatment groups. Detailed information on course enrollment transactions was collected from the University 
Registrar for all students in Chemistry 1 A. Because most students continue to adjust their class schedules 
during the first three weeks of the semester, we defined the enrollment baseline as the end of the third week. 

2. Results 

2a. Year 1, Student Performance 

Exam Scores 



We measured student performance in Year 1 by examining total course points earned out of 1,000 possible 
points. We looked specifically at the final exam scores (350 points possible) as the best measure of student 
performance. We chose this option because final exam scores have a large impact on the final grade, they focus 
on specific learning outcomes, and they are reflective of a consistent and standardized grading process. 12 

Of the 1,190 students who completed the course in Year 1, 672 (56%) consented to provide access to their 
course grades. No significant differences were found among treatment and control groups (Table 21), and we 
can assume that different experiences among the two groups had no effect on their performance in the course. 

Table 21. Comparison of Control and Treatment Groups by Final Exam Points (max = 350), Year 1 

Mean ± SD 





All 


Treatment Group 


Control Group 






(N=671)* 


(N=168) 


(N=503)* 


Effect size 


Year 1 fall semester 


236.5 ±51.8 


234.2 ±53.4 


237.3 ±51.2 


-0.059 



*Missing value = 1 



12 TAs scored exams as a group under the supervision of course instructors, and each TA was responsible for a specific question for all 
students. Other measures, such as overall course grades, included multiple, uncontrolled variables, such as lab grades (which were 
inconsistent across lab sections because TAs grading practices varied significantly among TAs). 
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Carry-Forward 

We were interested to know if specific learning outcomes differed among students in the control and treatment 
groups in Chemistry 1A, and if any difference persisted in the following semester (when there was no 
differentiation between the groups). Two outcomes were identified by instructors as important to a student’s 
success in the follow-up course: 1) Student demonstrates knowledge of the fundamentals of how molecules 
interact, and 2) Student demonstrates ability to apply the concept of equilibrium by describing how acids and 
bases interact to form stable/unstable molecules. 

Instructors also identified corresponding final exam questions from both Chemistry 1A and the subsequent 
course that addressed these learning outcomes. We compared student scores on exam questions to determine if 
there were any differences in conceptual understanding between the control and treatment groups. In order to 
identify possible differences between groups within each course and over time, we looked at the individual and 
combined scores of selected exam questions corresponding to identified learning outcomes (Table 22). 

These findings suggest that, for the two identified learning outcomes, the different experiences of students in the 
control and treatment groups did not have a significant impact on the students’ learning in Chemistry 1A. In the 
following semester of Chemistry, while results for individual questions varied, the overall means of the two 
groups indicate that students who had been in the Chemistry 1A treatment group performed slightly better, 
although the total effect size was still only 26 percent. 

Table 22: Comparison of treatment and control groups, 
by selected final exam question scores corresponding to learning outcomes 

Mean ± SD 



Course 


Learning 

Outcome 


Question 


Possible 

Points 


All 

(N = 272) 


Treatment 
Group 
(N = 66) 


Control 
Group 
(N = 206) 


Effect 

size 


Chemistry 1A 


1 


1 


13 


10.6 ±2.7 


10.2 ±2.8 


10.7 ±2.7 


-0.18 


2 


5 


4.0 ±2.0 


3.9 ±2.1 


4.0 ±2.0 


-0.02 


3 


5 


4.0 ±2.0 


3.9 ±2.1 


4.0 ±2.0 


-0.02 


2 


4 


5 


2.6 ±2.5 


2.5 ±2.5 


2.6 ±2.5 


-0.03 


5 


5 


2.0 ±2.5 


2.2 ±2.5 


1.9 ±2.4 


0.10 


6 


5 


2.6 ±2.5 


3.1 ±2.4 


2.5 ±2.5 


0.27 


Combined 


Total 


33 


21.7 ± 6.4 


22.0 ± 6.4 


21.7 ± 6.4 


0.05 


Subsequent 

Course 


1 


1 


2 


1.9 ±0.5 


1.8 ±0.6 


1.9 ±0.5 


-0.13 


2 


2 


1.3 ±0.9 


1.4 ±0.9 


1.3 ±0.9 


0.05 


3 


2 


1.1 ± 1.0 


1.2 ±1.0 


1.1 ± 1.0 


0.11 


2 


4 


2 


1.0 ± 1.0 


1.1 ± 1.0 


0.9 ± 1.0 


0.19 


5 


2 


0.9 ±0.6 


1.0 ±0.6 


0.9 ±0.6 


0.06 


6 


3 


0.6 ± 1.2 


0.9 ± 1.4 


0.5 ±1.1 


0.28 


Combined 


Total 


13 


6.8 ± 2.6 


7.3 ± 2.7 


6.6 ± 2.6 


0.26 



2b. Student Retention 

Table 23 summarizes retention data. Retention rates between Year 1 and Year 2 did not differ significantly. 
Additionally, there was no significant difference in completion rates between students in the control and 
treatment groups in Year 1. Of students in the control group, 94 percent completed the course; in the treatment 
group, 97 percent completed the course. 

Table 23: Course Retention Rates for Chemistry 1A, fall semesters only 





Year 1 


Year 2 


All Students 


Control 

Group 


Treatment 

Group 


All Students 


Number of students enrolled on Day 1 


1,215 


n/a 


n/a 


1,116 


Number of students enrolled after Week 3 


1,258 


971 


287 


1,202 


Number of students who completed the course 


1,190 


913 


277 


1,135 


Percentage of students who completed the course 


94.6% 


94.0% 


96.5% 


94.4% 
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D. Student Attitudes About the Course 

1. Data Collection 

To measure other possible changes in student learning, we examined the issues associated with learning that are 
not outcomes oriented (Erhmann, 1998). These issues included student access to the technology used in the 
course, their use of it, their opinions regarding using it, whether or not the students believe it impacted their 
learning, and how it affected their attitudes towards the course and learning. These issues were examined by 
using a combination of pre- and post-surveys and focus groups. 

Student Surveys. Table 24 summarizes the number of students responding to the surveys. Surveys addressed 
students’ perceptions of technology enhancements in the course. The on-line surveys also provided the 
opportunity for students to report about time spent on activities related to lecture and lab as well as some time 
spent on-line. All surveys included both close-ended and open-ended questions; responses to open-ended 
questions were later categorized and coded. 



Table 24: Chemistry 1A Student Survey Response Rates 





Year 1 

% 


N 


Year 2 

% N 


On-line pre-survey* 


16% 


198 


n/a 


In-class post-survey** 


76% 


904 


n/a 


On-line post-survey** f 


20% 


243 


57% 643 



*Total number of students enrolled at the time the pre-survey was administered in Year 1=1,258 
**Total number of students enrolled at the time the post-surveys were administered in Year 1=1,190 
t Total number of students enrolled at the time the post-surveys were administered in Year 2 =1,135 



Course Evaluations. We examined departmental course evaluations from Chemistry 1 A over the two years of 
the study and the previous year. We also examined departmental course evaluations from another popular large 
lecture course. Introduction to General Astronomy, whose instructor frequently over-enrolls the course and 
encourages students to watch lecture webcasts instead of attending class. These data provide us with some 
measure of comparison in student opinion regarding alternative uses of technology in large lecture courses. 

Student Focus Groups. A small number of focus groups in Y ear 1 were conducted for formative purposes, but 
had low student participation (N=13). 

2. Results 

2a. Year 1, Student Perceptions of Technology Enhancements 

Students provided feedback about the technology enhancements in Chemistry 1A including the on-line quizzes 
and assignments, in-class lecture slides, lecture webcasts, and the course website. A selection of student 
responses from Year 1 are presented in Table 25. Findings for Year 1 and Year 2 are differentiated. A 
summary of select student responses to the various surveys can be found in Appendix C. 

Year 1, On-line Quizzes & Assignments — Control vs. Treatment Groups 

Students in the treatment group had the opportunity to take each quiz three times 13 and keep their highest score. 
These students received immediate feedback on each attempt, including their score for each question. There 
were some differences between the opinions of students in the treatment group ( who had experience with on-line 
quizzes) and students in the control group (who did not) regarding doing activities on-line. According to the 
on-line post survey, students in the treatment group were pleased with their experience of taking quizzes and 
turning in pre-laboratory assignments on-line, and found these activities beneficial to learning chemistry. The 
majority of the students in the treatment group felt that completing assignments and quizzes on-line helped them 



13 Each quiz attempt randomly generated a new set of questions. 
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perform the tasks required in lab (70%) and made their lab time more productive (79%). Most (85%) also said 
that completing quizzes and assignments multiple times helped them learn Chemistry 1A concepts (Table 25). 

Responses from the Y ear 1 on-line post-survey also suggest that students generally liked the idea of doing more 
on-line work regardless of whether they were in the control or treatment group (Table 25). Most of the students 
(71%) would be willing to do more work on-line in order to spend less time in lab, with students in the treatment 
group (75%) more in favor than students in the control group (69%). A majority of students (70%) also said that 
they would be willing to do all of their homework on-line. Students in the treatment group (90%), who already 
had experience with completing quizzes and assignments on-line, were much more enthusiastic (significantly so 
with yj = 1 8.0, < 0.0001) about this prospect than were students in the control group (61%), who did not have 

the same experience. Students in the treatment group (87%) were also significantly (% 2 = 1 9.6 , p < 0.0001) more 
in favor of recommending courses with on-line assignments and quizzes than students in the control group 
(56%). 

The in-class post-survey, when compared to the aggregated responses in the on-line post-survey, showed a 
slightly higher positive response to these questions. Eighty-two percent of students responded that they would 
be willing to do more activities on-line and 75 percent indicated that they would be willing to do homework 
quizzes and pre-lab assignments on-line. 

Table 25: Student Attitudes about Chemistry 1A Technology Enhancements 
“Yes” Responses from Selected Year 1 On-line Post-Survey Questions 





All (N=218) 


Treatment 

Group 

(N=67) 


Control 

Group 

(N=151) 




% N 


% N 


% N 


“Would you be willing to do more activities on- 
line (e.g., quizzes, homework, discussion) so 
that you could spend fewer hours in the lab?” 


71% 154 


75% 50 


69% 104 


X 2 = 0.74, p= 0.39 


“If you had the choice, would you complete 
your homework quizzes and pre-lab assignments 
exclusively on-line?” 


70% 152 


90% 60 


61% 92 


X 2 = 18.0, p= <0.0001* 


“I would recommend that other students take 
courses like Chemistry 1 A that use. . .on-line 
quizzes or assignments.” 


65% 142 


87% 58 


56% 84 


X 2 = 19.6, p= <0.0001* 


“Did completing the pre-lab assignments and 
homework quizzes on-line help you perfonn the 
tasks required in lab?”** 




70% 47 






“Did taking quizzes or pre-lab assignments 
multiple times help you learn concepts about 
Chemistry 1A?”** 




85% 56 






“Do you think your time spent in lab was 
productive because you took the quizzes or pre- 
lab assignments on-line?”** 




79% 53 







* Statistically significant at the 99% confidence level 

** Questions specifically related to on-line quizzes and assignments were asked only of students in treatment groups 



Lecture Webcasts, Year 1 and Year 2 

We were particularly interested in student use of lecture webcasts and their relationship, if any, to attendance 
patterns and study habits. We report the survey results here. Detailed webcast usage statistics are reported in 
Section E (Student Use of Technology). 

In Year 1, students indicated through comments in the in-class post-survey that they watched the lecture 
webcasts because of the convenience and the ability to rewind and pause the lecture (e.g., “I can always rewind 
to listen to explanations if I didn't understand the first time.”). In Year 2, as in Year 1, the majority of students 
who watched lecture webcasts did so after missing a lecture. The second most popular reason was to study and 
review materials prior to exams. Only about one-quarter of students who responded to the on-line surveys in 
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both years chose to watch lecture webcasts instead of attending the lecture in person. Table 26 shows the 
distribution of responses. 



Table 26: Reasons Students Watch Lecture Webcasts 
Comparison of Responses to On-line Post-Surveys, by Year 





Year 1 


Year 2 


All 

(N=232) 


Control Group 
(N=164) 


Treatment Group 
(N=68) 


All 

(N=643) 


After missing a lecture 


66% 


65% 


69% 


72% 


To study and review material prior to exams 


63% 


65% 


56% 


41% 


In addition to reviewing notes/handouts 


n/a 


n/a 


n/a 


31% 


Instead of attending live lecture 


27% 


27% 


25% 


26% 


Instead of reviewing notes/handouts 


10% 


10% 


10% 


n/a 


Instead of seeking assistance from a TA or professor 


13% 


15% 


6% 


8% 


To prepare for lab 


4% 


3% 


6% 


n/a 


After attending a live lecture 


15% 


16% 


12% 


n/a 


Other 


7% 


7% 


7% 


5% 



Y ear 1 students who completed the on-line post-survey found the lecture webcasts useful, but preferred to attend 
lecture in person. Only a few students (12%) said that they would be willing to watch the lectures exclusively 
on-line; a majority (59%) indicated that they would never want that type of a class. Of the 904 students who 
responded to the same question on the in-class post-survey, more than 80 percent said they would not want that 
type of class. In Year 2, 86 percent of students reported they would not want to entirely watch lectures on-line 
instead of going to class. 

When students whose first language was not English (approximately 22% of the enrolled students) were asked 
about the utility of the on-line lectures, a large number (93%) responded that on-line resources were useful for a 
variety of reasons, such as the option to review difficult parts of the lecture on-line. 

2b. Year 1 and Year 2, Student Perceptions of the Course, Technology Enhancements, and Lectures 

It is an article of faith among educators and students alike that the large lecture format is not the best learning 
environment for students (e.g., Boyer Commission, 1999), although there are good data that suggest lectures 
serve many useful purposes for students and faculty (e.g., McKeachie, 1999). We were interested in learning: 

1) whether students have negative attitudes about the lecture experience, and 2) if the technology enhancements 
were correlated with positive perceptions students might have about the course. 

Student Perceptions of the Course. We first looked at the College of Chemistry’s course evaluation data. 
Students rate the course and the instructor on a scale from one to seven, with seven as the highest ranking. These 
data show that students’ ratings of Chemistry 1A in Year 1 and Year 2 were similar to the ratings given by 
students in the year prior to the beginning of the study, and consistently higher than the overall averages for 
lower-division courses within the Department (Table 27). 

Table 27: Student Course Evaluation Ratings for Chemistry 1A, by Year (fall semesters only) 





Previous Year 


Year 1 


Year 2 


Overall Course Rating, Chemistry 1 A 


5.81 


5.63 


5.61 


Average Rating for Lower Division Courses in the Department 


5.17 


5.17 


5.09 


Total Responses to Course Evaluations 


1,137 


1,215 


1,159 



Student Perceptions of the Technology Enhancements. Chemistry 1A students also responded very positively 
to the technical enhancements in our surveys and focus groups. One focus group participant commented, 

“I think I speak for a lot of people when I say that we’re glad we have this resource available to us.” Year 1 
focus group discussions and survey comments revealed additional student perceptions of technology 
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enhancements. One recurring theme was a reduction in student stress due to the availability of on-line 
resources. One student commented, “It took away a lot of the anxiety I had about the class, like if I would miss 
anything.” Another agreed, “It’s kind of like this little safety net.” 

In Year 2 we had a significant increase in the proportion of students responding to surveys. 14 When Year 2 
students (N=643 responses) were asked whether their experience with the technologies offered in Chemistry 1A 
affected their overall satisfaction with the course, 87 percent said “yes.” Of the 473 students that chose to 
explain their answer, 98 percent made positive comments about the technologies. Of those respondents with 
positive comments (N=467), 27 percent thought that the use of technology increased the availability and access 
to resources and helped them prepare for class. Nineteen percent felt that the implementation of the 
technologies improved the course. Another nineteen percent indicated that the technologies promoted learning 
and understanding of the course material. The remaining 32 percent (N=157) found the technologies generally 
helpful, useful, or convenient. 

As noted above, the majority of Year 1 student survey respondents would recommend that other students take a 
course that used technologies such as those offered in Chemistry 1A. Year 2 students were also asked if they 
would recommend that their peers take a course like Chemistry 1A that used various technologies such as the 
course website, lecture slides, on-line quizzes and pre-labs, lecture webcasts, and on-line office hours. A large 
majority (range 67-94% for all technologies except on-line office hours and lecture browser 15 ) said that they 
would recommend technology enhancements (Table 25). One hundred and seven students responded with 
detailed comments to this question. Of these, thirty-six percent commented that the technology was helpful and 
useful. Twenty-one percent either found that the use of technology in the course was an improvement or found 
that it increased their access to the resources in the course. Sixteen percent reported that the technology was 
convenient and accommodated their schedules more easily. Ten percent had general positive comments about 
the course and considered the technology a good supplement to their educational experience. 

Student Perceptions of the Lecture Experience. Overall, Y ear 1 and Y ear 2 students expressed the opinion that 
lecture webcasts were a valuable resource, but they considered webcasts most useful as a supplement to 
in-person lectures, rather than a substitute. Positive comments about the in-person lecture experience centered in 
three areas: interactivity (e.g., “easier to ask questions and talk to other students in the classroom.”); chemistry 
demonstrations (e.g., “I like to feel the heat from the demonstrations rather than feeling the heat from my 
monitor.”); and students’ personal discipline and concentration (e.g., “I concentrate better in lecture halls than 
my room”). The importance of interactivity of the in-person lectures may reflect the instructor’s uniquely 
engaging lecture style, and the use of the “ChemQuiz,” 16 a pedagogical technique for encouraging content-based 
peer-to-peer interaction. Table 28 outlines the details of student preferences. 

Table 28: Students’ Reasons for Attending Lecture in Person 
Year 2 Survey Comments, Coded (N=577) 



Student Comments 


Frequency 

% N 


Going to the lecture fosters personal discipline and concentration 


33% 


192 


The live lecture is interactive and facilitates conversations with other students 


23% 


135 


Attending lecture aids comprehension and learning 


15% 


85 


Prefer viewing the demonstrations in-person at lecture 


9% 


52 


Prefer attending lecture in person (no reason given) 


15% 


77 



14 High survey response rates in Year 2 can be attributed to a number of factors: the inclusion of minors in the study design, increased 
incentives for participation in surveys, or the fact that all students were required to complete weekly assignments and quizzes on-line 
(and therefore, had more opportunity to complete the survey). 

15 The BMRC Lecture Browser was not offered in Year 2, although archived lectures from Year 1 were available but not advertised. 

16 The “ChemQuiz” is an adaptation of a technique known as "Peer Instruction,” pioneered by Professor Eric Mazur, Department of 
Physics, Harvard University. Mazur’s technique is a method for teaching large lecture classes interactively. 
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E. Student Use of Technology 

1. Data Collection 

Website User Logs. We collected usage statistics for both years of the study. During Year 1 and Year 2, 
transaction logs were automatically generated by servers handling on-line materials in the following areas: on- 
line quiz taking patterns, general web page usage, and lecture webcast viewing. 17 In Year 1, we examined 
anonymous website usage for the most frequently used files. In Year 2, we were able to track the identity of 
each student along with other access statistics, and therefore have more detailed data on website usage because 
of authentication and a larger number of student consents. 18 We performed a transaction-log analysis for the 
following on-line materials: 

• Lecture webcasts, available live or as on-demand replays 

• On-line versions of slide presentations from lecture 

• On-line quizzes 

• On-line lab manual 

• Exam preparation materials, such as sample exams and exam solutions 

• Homework solutions 

• Communication tools, such as a bulletin board and chat tool 

• Course information, including the course lecture, laboratory, and exam schedule; homework 
assignments and due dates; grading rubrics; office hour times and locations; and contact information for 
instructors and TAs 

In analyzing website usage data, we selected two different metrics: the number of page hits (successful requests 
for non-image files) and the number of sessions (“a series of page requests by a visitor without 30 consecutive 
minutes of inactivity”). Because we authenticated all website users by requiring a login name and password in 
Year 2, our definition of sessions is even more robust than the industry standard. 19 

Lecture Webcasts. We examined the number of webcast views and the total time viewed by looking at 
transaction logs for the lecture webcasts. For comparison, we also collected statistics on general webcast usage 
patterns for other large lecture courses at UC Berkeley. 

Student Attendance. In an effort to better understand and compare patterns in student attendance for courses in 
which lecture webcasts were offered, we sampled the number of students present in the lecture hall, Pimentel 
Auditorium, for three courses in Year 2: Chemistry 1A (fall and spring semesters) and Introductory Physics 
(spring semester). See Table 29. While both chemistry courses offered lecture webcasts, the physics course did 
not. 



Table 29: Attendance Sampling by Course 



Course 


Year 2 
Term 


Days lectures 
offered 


Days attendance 
taken 


Percent 

sampled 


Students 

enrolled* 


Webcasts 


Chemistry 1A 


Fall 


41 


11 


27% 


1,202 


Yes 


Chemistry 1A 


Spring 


41 


12 


29% 


506 


Yes 


Introductory Physics 


Spring 


41 


12 


29% 


171 


No 



* Enrollment calculated at the end of the third week of classes. 



1 Due to a server logging error, for a three- week period during the middle of the Year 1 fall semester, the time of webcast views was not 
logged. For these missing data points, we imputed a value of time viewed based on the number of bytes transferred (which was 
successfully logged) and the average bytes-to-viewing-time ratio for other views from that IP address. 

18 We limited our analysis to students who completed the course, eliminating usage data from course staff and instructors, and from 
students who subsequently dropped the course. 

19 The Internet Ad Bureau (1997), which defines the industry standard for web usage measurement, suggests several different metrics. 
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2. Results 

Unless otherwise noted, we will report general website usage patterns for Year 2. (A detailed analysis of 
transaction logs, student background, student performance, and student attitudes is the subject of another 
paper.) Details of webcast statistics, and their relationship to other measures, are reported at the end of this 
section. 

2a. Website Usage Patterns 

Year 1 General Website Usage 

In Year 1, when the website was accessible to the general public, the main course website received 690,916 
page hits for the semester — an average of 581 page hits per student. The bulk of the overall website usage was 
by UC Berkeley users; more than 85 percent of page requests originated from users in the berkeley.edu domain. 
Seventy-eight percent of requests came from computers in the campus residence halls. Because so much of the 
traffic originated on-campus, and in the residence halls in particular, it is reasonable to assume that a vast 
majority of the website usage resulted from students enrolled in the course. 

Year 2 General Website Usage 

Students who completed the course showed a higher level of website use than in Year 1 for both required 
(e.g., weekly quizzes and assignments) and optional portions of the website. Table 30 shows the level of usage 
per week of different resources on the website. Table 3 1 further breaks down usage to show the distribution of 
number of sessions for each type of resource. It is interesting to note that some resources (such as exam 
materials) were used relatively uniformly by many students, while the usage of other materials (such as lecture 
slides) varied much more between students. 



Table 30: Summary of Year 2 Web Usage by Type of Resource 
(N=l,135 students who completed the course) 





Page hits per 
student 


Sessions per week, 
per student 


Percentage of students 
who used the resource at 
least once per week 

(> 1 5 sessions per semester) 


Total website 


2,745.52 


8.65 


100.0% 


Course information 


292.71 


6.38 


99.9% 


Quizzes 


701.15 


2.16 


98.0% 


Lecture slides 


795.53 


1.33 


49.7% 


Exam materials 


142.92 


1.32 


61.9% 


Lab manual 


68.83 


1.08 


43.7% 


Communication tools (bulletin board, chat) 


73.62 


0.78 


18.3% 


Homework solutions 


36.89 


0.49 


16.0% 




Views per 
student 


Average length of 
each view 

(hh:mm:ss) 


Percentage of students 
who viewed webcasts at 
least once per week 


Lecture Webcasts 


13.02 


00:22:28 


25.37% 
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Table 31: Summary of Year 2 Website Usage, Total number of sessions for each type of resource 
(N=797 students who consented and completed the course) 





0 

Sessions 


1-5 

Sessions 


6-10 

Sessions 


11-15 

Sessions 


16-20 

Sessions 


21-25 

Sessions 


25+ 

Sessions 


Total website 


0 (0%) 


0 (0%) 


0 (0%) 


0 (0%) 


0 (0%) 


0 (0%) 


797 (100%) 


Course infonnation 


0 (0%) 


0 (0%) 


0 (0%) 


0 (0%) 


1 (0.1%) 


6 (0.8%) 


790 (99.1%) 


Quizzes 


0 (0%) 


0 (0%) 


3 (0.4%) 


8(1.0%) 


29 (3.6%) 


106 (13.3%) 


651 (81.7%) 


Lecture slides 


3 (0.4%) 


150(18.8%) 


121 (15.2%) 


93(11.7%) 


98(12.3%) 


69 (8.7%) 


263 (33.0%) 


Exam materials 


0 (0%) 


19 (2.4%) 


95 (11.9%) 


153(19.2%) 


156(19.6%) 


126 (15.8%) 


248 (31.1%) 


Lab manual 


0 (0%) 


54 (6.8%) 


175 (22.0%) 


195 24.5(%) 


131 (16.4%) 


88(11.0%) 


154(19.3%) 


Communication tools' 


31 (3.9%) 


387 (48.6%) 


153(19.2%) 


48 (6.0%) 


38 (4.8%) 


34 (4.3%) 


106 (13.3%) 


Homework solutions 


113(14.2%) 


282 (35.4%) 


157(19.7%) 


104(13.0%) 


77 (9.7%) 


39 (4.9%) 


25 (31%) 


Webcasts 2 


25(3.1%) 


350 (43.9%) 


117(14.7%) 


97 (12.2%) 


49(6.1%) 


33(4.1%) 


126 (15.8%) 



1 bulletin board, chat 

2 number of views 



Website usage over time. Website usage varied a great deal at different points in time. Figure 1 shows usage of 
different on-line resources, except lecture webcasts, 20 over time in Year 2. Lecture webcast usage is displayed 
in Figure 2. Overall, the most noticeable pattern is a weekly cycle, with the heaviest use on Sundays and 
Mondays, and little to no use on Fridays and Saturdays. In addition, total usage peaks near the dates of midterm 
and final exam s . 

Different on-line resources have different patterns of use. Usage of lecture slides, exam materials, webcasts and 
course information are closely related to the exam schedule. Usage of communication tools, lab manual, and 
homework solutions, on the other hand, hold relatively constant throughout the semester. Usage of on-line 
quizzes follows a weekly cycle, with peaks just before the Monday deadline for homework quizzes and no 
connection to the exam schedule. 

2b. General Lecture Webcast Usage 

In Year 1, seven percent of requests for videos were for live lectures, and the remainder (93%) were for archived 
videos. Each Chemistry 1A lecture webcast was viewed approximately 0.355 times per enrolled student. 
Duration of each play was short. Over 60 percent of replays lasted less than ten minutes, and only ten percent of 
replays lasted for the entire lecture. Webcast usage peaked during the 4th, 8th, 12th, and 16th weeks of the 
semester, corresponding exactly to the course’s midterm and final exams. In general, students accessed the 
archive beginning around 10:00 a.m. in the morning, building to a peak around noon. Usage continued heavy 
throughout the afternoon and evening with a short drop around dinnertime. Usage continued until 2:00 a.m. 
when it fell off rapidly until it picked up again later that morning. 

In Year 2, five percent of requests for videos were for live lectures; each Chemistry 1A lecture webcast was 
viewed approximately 0.317 times per enrolled student, and each Chemistry 1A student viewed an average of 
7 hours and 26 minutes of webcasts during the semester. 21 Most webcast views were short; for Chemistry 1A, 

43 percent of all views were shorter than 10 minutes. Only 20 percent of views were 50 minutes or longer (the 
length of an entire lecture). Similar to Year 1 data and student survey comments, these observations of webcast 
usage suggest that students in general are not using webcasts to view entire lectures, but instead are using them 
on an “as-needed” basis, to review or study specific segments of the lecture. Figure 2 shows that in Year 2, 
webcast usage peaks in weeks 4, 8, 12, and 16 — immediately prior to each of the four exams. This pattern 
clearly indicates that students use webcasts, along with lecture slides and exam materials, primarily to study and 
review for exam s . 



Webcast statistics were in a different metric. 

21 Webcast usage for Year 2 was lower than Year 1. This may have been related to the fact that the BMRC Lecture Browser was not 
implemented. 






Figure 1: Website Usage Over Time 
(Chemistry 1A, Year 2) 
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course week 








Figure 2: Webcast Viewing over Time 
(Chemistry 1A, Year 2) 




course week 
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Lecture Webcast Usage: Chemistry 1A Compared to Other Courses 



Year 1. Our data support the observation that students use webcasts primarily for on-demand study, rather than 
replacing attendance during live lectures. These observations paralleled trends in other courses using the lecture 
webcast service in the Year 1 fall semester (N=21) and spring semester of the previous year (N=13). 



Figure 3 shows the number of plays of all webcast lectures each week during Year 1 at UC Berkeley. A class 
with two midterms typically schedules them during weeks five and ten, and a class with one midterm typically 
schedules it between weeks eight and eleven. Week sixteen corresponds to the final examination period on 
campus (Rowe et. al., 2001). 

Figure 3: Plays Per Week of all UC Berkeley’s Webcasted Courses, Year 1 




Spring 2000 Fall 2000 



Year 2. In comparison to webcast viewing in other large introductory lecture courses in both the fall and spring 
semesters of Year 2, students taking Chemistry 1A had a slightly lower lecture webcast usage per student than 
some other courses (Table 32). It is interesting to note the large variation in webcast usage levels between 
different courses. It is also interesting that usage in some courses is more closely related to exam schedules than 
in other courses (Figure 4). Because webcasts are available to the general public and are accessed anonymously, 
it is difficult to know whether these differences are due to enrolled students or to non-student access. 

Table 32: Lecture Webcast Viewing, Large Lecture Courses, Year 2 



Course 


Year 2 
Term 


Enrollment 


# of 
lectures 


Total # of 
views 


% of live 
views 


# student 
views per 
lecture 


Viewing 
time per 
student 


Introduction to General Chemistry (Chem 1A) 


Fall 


1135 


41 


14,734 


4.8% 


.317 


7:26 


Introduction to Computer Programming (CS 61 A) 


Fall 


414 


42 


9,039 


3.6% 


.520 


11:57 


Introduction to General Astronomy (Astro 10) 


Fall 


990 


40 


25,736 


3.3% 


.650 


18:19 


Introduction to Computers (IDS 110) 


Fall 


426 


37 


5,487 


11.1% 


.348 


6:12 


Introduction to Human Nutrition (NS 10) 


Fall 


670 


24 


10,693 


3.6% 


.665 


9:50 


General Biology (Bio IB) 


Spring 


467 


42 


33,249 


1.4% 


1.695 


31:03 


Introduction to General Chemistry (Chem 1 A) 


Spring 


506 


41 


8,070 


2.8% 


.389 


10:16 


Introduction to Computer Programming (CS 61 A) 


Spring 


311 


44 


10,295 


8.1% 


.752 


21:40 


Introduction to Computers (IDS 110) 


Spring 


388 


36 


14,931 


2.9% 


1.069 


8:11 


Introduction to Human Nutrition (NS 10) 


Spring 


695 


44 


19,750 


1.7% 


.646 


14:25 








Figure 4: Webcast Viewing over Time 
Large Introductory Courses (Year 2 Fall Semester) 




course week 
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Viewing over time. The most notable trend for the majority of the courses we examined is spikes in lecture 
webcast viewing immediately preceding exams (Figure 4). For many courses, viewing jumped to three- to four- 
times the typical daily usage in the days immediately preceding exams. Viewing then dipped to almost zero in 
one to three days following the exam. This post-exam dip is noticeable even in courses without a pre-exam 
spike. The strong correlation with exam schedules reinforces our interpretation that students use the webcasts 
primarily as a tool for studying and review, rather than as a regular substitute for in-person lecture attendance. 

Pre-exam spikes, and webcast viewing in general, tended to increase over the course of the semester (in spite of 
the fact that course enrollment tends to erode slightly, as students drop the course over the course of the 
semester). Since most of the students in these courses are first-year students at the university, or at least new to 
the subject matter of the course, these trends may mean that students are adjusting to their first year of college 
and learning how to use the technology resources available. 22 As the semester progresses, more students may 
become comfortable with the webcast technology or find it a useful tool for their own study habits. 

2c. Lecture Webcast Viewing and Attendance 

There has been some consternation on campus that the presence of lecture webcasts results in decreased 
in-person classroom attendance. Although the majority (>80%) of students in Chemistry 1A reported that they 
would not substitute remote viewing for attending lecture in person, but prefer to use webcasts as a study aid, 
other data suggest that students are opting out of attending all three lectures per week. Thirty-one percent of 
students report in the Year 2 on-line post-survey (N=243) that they attend lecture less than three times per week 
and 25 percent stated that they replace the lecture with webcasts. 

In Y ear 2 we conducted student attendance counts for a random sample of lectures in the fall and spring 
semesters. Attendance at the sampled lectures in the fall semester ranged between 90 percent (near the 
beginning of the semester) to 68 percent (near the end). In the spring semester, attendance at Chemistry 1 A 
lectures ranged from 82 percent to 60 percent. 23 In comparison, student attendance at lectures for a section of 
Introductory Physics in the spring semester showed a similar trend (Table 33), but a much larger drop-off in 
attendance. For this course, where webcasts were not used, attendance was high in the first half of the semester 
but dropped steadily, from approximately 90 percent of enrolled students attending lecture in the fourth week of 
the semester to approximately 60 percent attending by the seventh week. In the second half of the spring 
semester, however, student attendance dropped dramatically, averaging about 45 percent of enrolled attending 
lectures in the last few weeks of the semester. 

Table 33: In-person Attendance at Lectures, Comparison Across Courses, Year 2 



Chemistry 1A, fall semester 

(Enrollment = 1,202) 


Chemistry 1A, spring semester 

(Enrollment = 506) 


Introductory Physics, spring semester 

(Enrollment =171) 


Week 


Lecture 


Attendance 


% 


Week 


Lecture Attendance 


% 


Week 


Lecture Attendance 


% 


3 


7 


1,024 


85% 


6 


13 


426 


82% 


6 


13 


157 


92% 


3 


8 


930 


77% 


6 


14 


393 


78% 


6 


14 


138 


81% 


4 


Mid-term 1 






6 


15 


370 


73% 


6 


15 


123 


72% 


7 


19 


913 


76% 


8 


Mid-term 1 






7 


Mid-term 1 






7 


20 


894 


74% 


9 


21 


367 


73% 


9 


21 


116 


68% 


8 


21 


949 


79% 


9 


22 


322 


64% 


9 


22 


100 


58% 


8 


Mid-term 2 






9 


23 


249 


49% 


9 


23 


61 


36% 


9 


24 


875 


73% 


11 


27 


339 


67% 


11 


27 


101 


59% 


11 


31 


813 


68% 


11 


28 


365 


72% 


11 


28 


79 


46% 


11 


32 


762 


63% 


12 


Mid-term 2 






12 


Mid-term 2 






12 


Mid-term 3 






12 


30 


365 


72% 


13 


34 


74 


43% 


14 


36 


769 


64% 


14 


35 


332 


66% 


14 


35 


76 


44% 


14 


37 


794 


66% 


14 


36 


335 


66% 


14 


36 


77 


45% 


15 


39 


800 


67% 


14 


37 


312 


62% 


14 


38 


77 


45% 



22 The on-campus Student Learning Center, which works with approximately 35 percent of Chemistry students in the fall course, reported 
that since the implementation of the technology enhanced course, students often ask how they can develop strategies for navigating all of 
the on-line resources available. 

23 With the exception of March 22, 2002 — the day before Spring Break — which had approximately 50 percent attendance. 
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The similarity in attendance patterns between the two chemistry courses, where webcasts were available, and the 
physics course, where webcasts were not available, suggests that the drop in attendance throughout the semester 
is not an unusual phenomenon. Anecdotal reports from other courses also support this conclusion. The steep 
attendance drop-off in the physics course — steeper than in the two chemistry courses with webcasts — indicates 
that student absences are the result of a more complicated interaction of factors than simply the existence of 
webcasts, such as time of day and style of lecture. 

Finally, it should be noted that in the fall semester of Year 2, the total lecture hall seating capacity for 
Chemistry 1A was 1569 (3 lectures/day x 523 seats). In fact, at no time was that capacity approached in our 
attendance counts (N=l 1 counts; range=762 to 1024). Apparently all students attending lectures could have 
been accommodated in two lectures instead of the three that were allocated. 

The relationship between webcast viewing and lecture attendance 

For the two courses for which lecture webcasts and attendance counts were available (the fall and spring 
semesters of Chemistry 1A in Year 2), we performed a correlation analysis to determine if there was any 
relationship between attendance at a specific lecture and viewing of that lecture’s webcast. For the fall semester 
course, lower attendance for a lecture was correlated with greater webcast viewing of that lecture, although this 
correlation was not quite statistically significant (R= -0.58, p= 0.060). For the spring course, there was no 
significant correlation between lecture webcast viewing and attendance (R= 0.15, p= 0.65). This suggests that 
students are not using webcast viewing as a substitute for in-person attendance in a consistent, large-scale way. 
In both cases, the date of the lecture relative to exam schedules was a much stronger predictor of attendance, 
with a steady decrease in attendance observed throughout the semester. 

It should be noted that an exceptionally popular course on campus, Introduction to General Astronomy, 
over-enrolls by approximately 90 percent of the seat capacity in Pimentel Auditorium. The instructor allows all 
interested students to enroll, and encourages them to substitute lecture webcast viewing for in-person 
attendance. A review of departmental course evaluations for this course indicates that the overall ratings for the 
course and the instructor are remarkably high, and very few students complain about the lack of seating 
capacity. Transaction log analysis indicates that this course gets an exceptionally high level of lecture webcast 
viewing (see for example, Figure 4 and Table 32). 

2d. Web site and webcast usage in relation to student performance and attitudes 

In Y ear 1 and Y ear 2, we had the opportunity to further investigate the relationship between usage of on-line 
resources and both observed and reported student behaviors such as grades and attitudes about study habits. 

Student Lecture Webcast Viewing and Performance. Year 1 

Although we were unable to track actual webcast viewing of individuals in Y ear 1 , we were able to identify 
self-reported webcast viewing, individual attitudes, and student performance for a group of students. As 
reported above, Y ear 1 students reported their use of on-line lecture webcasts on the on-line post-survey 
(N=243). Of these students, 147 also consented to provide access to their course grades. We performed simple 
regression analyses to model the effect of webcast viewing and reasons for viewing on students’ course grades. 
Overall, self-reported webcast usage was related to lower course grades (although this relationship was not 
statistically significant). Students’ reasons for using on-line webcasts showed a weak relationship with student 
grades in this sample. Students who reported using lecture webcasts as a replacement for the in-person lecture 
had lower scores in the course overall. Since webcast users were a self-selected group, students who used 
webcasts as a replacement for in-person lecture attendance may simply have had poor study habits. Table 34 
illustrates our regression analysis of how student webcast usage and reasons for usage (according to self-reports 
on Year 1 survey) affected their overall course grades. 
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Table 34: Students’ Reasons for Watching Lecture Webcasts, 
Year 1 On-line Post-survey Responses (N=243) 





Mean ± SD 


Correlation to overall 
course grade* 


Frequency of webcast usage (self report) 


1.2 ±0.8 
views per week 


P= -25.7, R 2 =0.02 
per view per week 




Frequency 


Correlation to overall 




% (N) " 


course grade* 


To study and review material prior to exams 


76% (N=136) 


P= -20.7, R 2 =0.007 


To prepare for lab 


5% (N=9) 


(3= -41.3, R 2 =0.008 


Instead of going to lecture (live) 


33% (N=59) 


P= -55.5, R 2 =0.06** 


After missing lecture (archived) 


79% (N=146) 


P= -46.7, R 2 =0.03** 


After attending live lecture 


18% (N=31) 


P= 22.0, R 2 =0.006 


Instead of reviewing notes/handouts 


12% (N=2 1 ) 


P= 31.2, R 2 =0.01 


Instead of seeking assistance from TA or professor 


15% (N=26) 


P= —31.9, R 2 =0.01 



* a negative p indicates that the factor contributes to a reduced point total 
** statistically significant at p = 0.05 



Student Lecture Webcast Viewing, Website Usage, Background, and Performance. Year 2 

In Year 2, the high participation rate of consenting students allowed us to analyze variables that might link use 
of on-line resources, performance, and background. We collected an immense amount of data on student web 
usage, not all of which is reported here. In brief, a multiple regression analysis found that four background 
variables are significantly related to performance: females score lower than males in the course, and higher 
SAT I math scores, SAT II scores, and high school GPAs are associated with higher course grades. No other 
background variables were statistically significant. We also found that usage of the course website was 
significantly related to higher course grades. Webcast viewing, on the other hand, had no significant effect on 
course grades. Students’ reasons for viewing webcasts, however, had a large effect on course grades; students 
who used webcasts as a replacement for live lectures on average scored lower in the course. Details of this 
analysis are below. 

We performed a multiple regression analysis to estimate the effect of students’ usage of on-line resources on 
their course grades. Course grades were measured on a 1,000-point scale, with the difference between a B and 
an A equal to about a 120-point increase and the difference between a B and a B+ equal to about a 60-point 
increase. In our regression model, we included students’ amount of website usage (excluding webcasts), 
measured as the number of sessions per week; and their amount of webcast viewing, measured as the number of 
hours of viewing per week. We also included students’ self-reported reason for viewing webcasts, as indicated 
on the on-line post-survey: those who viewed webcasts instead of attending live lecture (referred to as 
“replacers”) versus those who did not. 

We also included in the model measures of students’ demographic background (gender and ethnicity), 
socioeconomic background (parental income and education), and educational background (SAT I and II scores 
and high school GPA), to control for their effects. The total R 2 of this model was 0.51, indicating that these 
variables together account for more than half of the variability in course grades. The adjusted R 2 of the model 
decreases from 0.50 to 0.46 with the removal of the “replacer” variable and to 0.44 with the removal of the 
usage measures, indicating that each of these provides appreciable additional predictive power. 

In the total model (Table 35), four background variables were significant at the 95 percent confidence level: 
females scored twenty points lower than males in the course, and higher SAT I math scores, SAT II scores, and 
high school GPAs were associated with higher course grades. No other background variables were statistically 
significant. 
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Student usage of the course website was significantly related to higher course grades: an increase of one session 
per week was related to a 3.8-point increase in course score. Lecture webcast viewing, on the other hand, had 
no significant effect on course grades. Interestingly, students’ reasons for viewing lecture webcasts made a big 
difference in course grades: students who used webcasts as a replacement for live lectures on average scored 
31 points lower in the course — half the difference between a B+ and a B. 

We also repeated this regression breaking out the usage of individual portions of the course website. The 
adjusted R 2 of the model increased by less than one percent. Usage of only one portion of the website was 
statistically significant: the use of on-line exam materials was associated with higher course grades 
(P= 15.8 ± 6.3 course points per session per week; p= 0.013). 24 Trends in the other variables remained 
unchanged. 



Table 35: Predicting Student Course Points: Ordinary Least Squares Regression 
R 2 =0.51, N=483 students with all data available 



Variable 


Parameter Estimate (p) 
(estimate ± std err) 


p-value 


* Female gender 


-20.4 ±7.1 


0.004 


White ethnicity 


-0.05 ±7.5 


0.995 


Parental education 


-0.18 ± 1.3 


0.89 


Parental income (logarithm) 


-1.0 ±3.4 


0.77 


SAT I verbal score 


-10.1 ± 5.8 per 100-point increase 


0.08 


* SAT I math score 


41.4 ± 7.4 per 100-point increase 


<0.0001 


* SAT II combined score 


18.3 ±3.3 per 100-point increase 


<0.0001 


* High School GPA 


51.2 ±14.4 


0.0004 


Website usage (sessions per week) 


3.8 ±0.8 


<0.0001 


Webcast usage (viewing hours per week) 


1.3 ±4.7 


0.78 


* “Replacer”** 


-30.6 ±8.0 


0.0002 



* Statistically significant predictor of course grade (p < 0.05) at the 95% confidence level. 

** Students who self-reported that they viewed Chemistry 1A webcasts instead of attending live lectures. 



24 Two other variables were significant at the 90 percent confidence level: increased usage of lecture slides predicted higher course 
grades (p= 5.9±3.3; p= 0.076) and increased usage of the on-line lab manual predicted lower course grades (p= -9.2±5.3; p= 0.080). 
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IV. DISCUSSION 

A. Overview of Cost Data 

Although the technology enhancements increased the cost of Chemistry 1A in the initial pilot year, these 
costs are a relatively small percentage of the total cost of the course. In the first year, developing 
technology-enhanced materials added $68,731 (7.1 percent of total course costs) to the development/revision 
costs. Three-quarters of that additional development cost was recovered in course delivery cost savings in the 
first year that the technology-enhanced course was offered. If instructors reuse the technology-enhanced 
products created for Y ear 1 in subsequent semesters, then the course development cost will decrease 
substantially. In fact, we found that the cost of developing technology-enhanced materials dropped to less than 
three percent of total course costs in Year 2, and that the investments in technology-enhanced materials paid for 
themselves in reduced course delivery costs over the two course offerings (one offering each in Years 1 and 2). 

Development costs (web site, lecture slides, online quizzes) would decrease in future years if instructors were to 
revise or reuse existing digital or multimedia products in their course. The two largest development and revision 
costs were for the preparation of the course web site and for the preparation of the lecture slides for the lectures. 
Development costs decreased in Year 2 by almost 70 percent and we expect the same in future years, with 
instructors revising or reusing existing digital or multimedia products in their course. The degree to which reuse 
of the technology enhancements by other faculty will occur is not clear, as the introductory chemistry course at 
UC Berkeley is taught on a rotating basis by tenure-track faculty who are active researchers. The result is that 
each faculty member has a distinct philosophy, strong preferences, and considerable flexibility in how to teach 
the class. 

B. Faculty and Teaching Assistant Time 

Instructors spend less time doing repetitive tasks in the technology-enhanced version of Chemistry 1A. 

Specifically, our data show that instructors spend considerably less time preparing for class since the 
introduction of the lecture slides. The lead instructor for Digital Chemistry 1A estimated an average time- 
savings of 53 percent overall due to technology enhancements to the course. This estimate included 35 percent 
time-savings in lecture preparation. The cost savings are considerable, and can be captured each year with only 
minor revisions in subsequent years. In the traditional course, instructors spent several hours each day of lecture 
creating the chalkboards. In the technology-enhanced course, instructors are freed from this time-consuming 
task because they have created the lecture slide presentations before the beginning of the semester. It should be 
noted that students were particularly fond of the online lecture slides as study aids, and this was reflected in the 
heavy use of these resources. 

Instructors spend less time answering routine questions in the technology-enhanced course because 
students are able to find the necessary information online. Instructors spend approximately 50 percent less 
time answering routine questions about the course, including time spent in office hours. More than 60 percent 
of students report that they go to the web site rather than teaching staff office hours to get answers to questions 
at least some of the time. Rather than spend less time on the course overall, instructors report that they spend 
the saved time on other activities related to instruction and course development. 

Teaching Assistants are relatively inexperienced teachers and spend a large amount of their time at the 
start of the semester negotiating the varied responsibilities of being a TA, not using technology to enhance 
their teaching. Few of the TAs had graduate level teaching experience. Although the majority of TAs come 
into Chemistry 1A with access to and experience using educational technologies (aside from online office hours 
and lecture webcasts), few find that the technologies are central to their teaching. TA surveys indicate that, by 
some measures, they were more comfortable with the technologies and the benefits provided as the semester 
progressed (e.g., webcasts, perception that technologies saved time and freed up time in lab). 

The TAs in the treatment group spent less time grading, and appeared to spend less time on 
administrative tasks both in and out of the classroom. TA administrative time is saved in class because of 
the online pre-lab resources. Based on our observations, TAs in the treatment group did fewer administrative 
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tasks in lab. There appeared to be a significant time-savings in grading as well. The availability of online of 
quizzes that were automatically graded reduced the time that TAs spent grading, which is a task most of them 
find menial. More than 80 percent of TAs surveyed were willing to migrate these tasks online. An interesting 
finding was that TAs and students in the treatment group appeared to spend more time on the Discussion and 
Experiment in laboratories, and that most students in treatment groups felt that they were never rushed. 

Because TA salaries and benefits are 60 percent of all course costs for Chemistry 1A, reducing, or at least 
reallocating, TA time presents opportunities for saving money, serving more students, and/or redistributing TA 
time to allow for richer interactions with students. By freeing TAs of tasks that they considered menial and 
burdensome, the technology enhancements allowed TAs to increase the time they spent doing other instructional 
activities both inside and outside of the classroom. For example, newly available time appeared to result in 
more time for other activities (e.g., conducting the experiment or increased TA-student interaction) rather than 
in less time spent in lab section meetings. Year 2 data confirmed that TAs, especially after they had gained 
familiarity with the technology enhancements, saw the technology more as a time-saver than as a way to foster 
increased student understanding of the course material. 

C. Student Performance and Attitudes 

Student performance was not significantly affected by the technology enhancements in the Year 1 
experiment. We have found no significant difference between students in the treatment and control groups in 
grades, retention, or their conceptual understanding in the following semester of chemistry. However, the 
intention behind introducing technology enhancements to Chemistry 1 A was to do no harm, not necessarily to 
raise course grades. Both years of data indicate that those students who used webcasts the most frequently 
(based on self-reported or observed data) had poorer final grades. Multiple hypotheses may explain this result 
(e.g., low-performing students are more likely to rely on webcasts as back-up, webcasts actually impede 
performance). Year 2 data indicate that student use of webcasts as a replacement for the in-person lecture 
attendance results in poorer final grades, but that webcast usage for other reasons does not have the same 
negative effect. Students who used the course web site more often, on the other hand, tended to have better 
course grades. 

Students find the technologies to be an exceptionally positive component of the course. A little explored 
topic in cost-effectiveness studies is the impact on student “costs” (i.e., what do students perceive as 
benefits/costs of the technology?). Our attitudinal data collected over two years suggest that students perceived 
the suite of enhancements as a significant contributor to their overall satisfaction with this large lecture course, 
web usage data, when triangulated with performance and attitudinal data in Y ear 2, suggest that students use the 
online enhancements: 1) on an “as needed” basis, 2) as a significant resource in their study strategies, especially 
when preparing for exams, and 3) as safety nets for their individual circumstances (e.g., disabilities, English 
proficiency, personal schedules). Of the almost 500 students who wrote in comments on surveys, 98 percent 
thought that the use of technology increased the availability of and access to resources, helped them prepare for 
class, improved the course, promoted learning and understanding of the course material, and/or were either 
helpful, useful, or convenient. By a significant amount, students in the treatment group responding to the online 
survey in Year 1 wanted more online assignments and were more likely than their counterparts in the control 
group to recommend this type of course to other students. 

Lectures can be a positive draw for students. It is an article of faith among educators and students alike that 
the large lecture format is not the best learning environment for students (e.g., Boyer Commission, 1999), 
although there are good data that suggest lectures serve many useful purposes for students and faculty (e.g., 
McKeachie, 1999). Our findings from Chemistry 1A and an introductory astronomy course show that excellent 
lectures presented by dynamic teaching staff are a huge draw for students. In Chemistry 1A, reported reasons 
for attending the lectures included interacting with other students and the instructors, experiencing live 
demonstrations, and encouraging personal discipline and concentration. Many students alluded to the positive 
social benefits of participating in an “event” with large numbers of other students. 
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A large number of students regularly do not attend lectures. At no time was full lecture hall capacity 
(N=1569) approached in our attendance counts (range=762 to 1024). In Year 2, 31 percent of survey 
respondents report attending lecture less than three times per week and 25 percent report replacing the lecture 
with webcasts. Attendance data on another introductory science course, which did not use webcasts, indicate 
that webcasts alone are not the reason for decreased student attendance at lectures. Comparative attendance and 
viewing data from other courses that used online video lecture archives at UC Berkeley in Year 1 (Rowe et. al., 
2001) and Year 2 suggest that the degree to which students opt out of attending lectures may be heavily 
influenced by time of day (e.g., early morning) and the style of lecture delivery. 

D. Prospects for the Reuse of Space and Time in Lectures and Labs 

The availability of the on-demand replays of lectures has the potential to allow a larger number of 
students to be enrolled in the course. Our data indicate that most students in Chemistry 1A use the online 
lectures primarily as study aids, and the majority (>80 percent) would not substitute remote viewing for 
attending lecture. However, students still report, and we observed, that they do not attend lecture the “required” 
three days per week — but rather closer to an average of two days per week. Our data on actual lecture 
attendance confirm what many instructors already know, that a large number of students do not attend lecture on 
a daily basis. 

Reduction in the number of lectures given each day from three to two (or one) — perhaps by requiring 
some students to attend lectures virtually — could realize appreciable saving in faculty time devoted to 
lecture as well as free up lecture hall space for other courses. Because the same lecture is given three times 
per day, staff and facilities costs could be saved if a portion of students either opted out of attending lectures, or 
a lottery system were devised so students were required to view a certain number of lectures per semester 
online. Our data indicate that the lecture hall is not filled to capacity, and one lecture per day could easily be 
eliminated. 

Time spent in laboratory sections could hypothetically be reduced. Based on our observations, average time 
spent on experiments and discussion combined was approximately three hours instead of the four hours allotted 
for these activities. There were always students who straggled in lab, however, and filled up the full time 
allotted for the lab. If a time reduction proves practical, Chemistry 1 A could add approximately 20 lab sections 
per week and accommodate approximately 600 students without acquiring new space for labs. While more TAs 
would need to be hired to teach additional sections, no additional costs would be incurred for new facilities in 
this scenario. Although our observations and TA self-report data show that a reduction of lab time from four to 
three hours is possible, it is not probable. After providing the opportunity to conduct labs in three hours, we 
found that the four-hour section seems to be the desired interval for the activities that take place in lab, which 
include not only the experiment but formal discussions and informal one-on-one interaction among students and 
TAs. 
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V. CONCLUSION 

Our study provides some intriguing data on both the costs and utility of the current technology enhancements in 
a large lecture course at a major public research university, which we discuss below. 

A. Challenges to Robust Research 

The challenges associated with executing a robust research analysis of a “fast-running” experiment of this scope 
are substantial. The size and complexity of the Chemistry 1A teaching and learning environment and its 
placement within an even larger and more complex public research university cannot be overemphasized. 
Implementation and evaluation of large-scale experiments of this sort require not only robust campus technology 
support structures, but the gathering of different types of data (costs, learning outcomes, transaction log 
statistics) from disparate campus units and individuals (institutional, faculty, staff, students, etc.). There are 
many obstacles to navigate. For example: 

• Maintaining a balance between good research design and not disrupting the teaching of a large introductory 
course; 

• Gathering consents from more than 1,000 students per semester (25 percent of whom were under 18); 

• Ferreting out Activity Based Cost data that were distributed among many units on campus and in different 
formats; 

• The merging of key technology support units in Year 2 of the study compounded the difficulty of getting 
reliable cost data; 

• Inconsistencies and performance problems in commercial LMS software (e.g., quizzing tool). 

• Campus cultures and habits that were not always sympathetic to the demands of experimentation and 
research in real time, large introductory courses; 

• Constant editing of the course by faculty and staff, which ultimately benefits student learning and is the 
sine qua non of good teaching, but makes controlling variables in an experiment of this sort exceptionally 
difficult. 



B. The Importance of Convenience and Choice for Students 

Given that large lecture courses have a reputation among educators as being poor learning settings, especially 
among those who advocate a predominantly student-centered approach to learning, our data show that students 
were both exceptionally enthusiastic about the lecture component of the course and engaged with the online 
materials. Survey responses and transaction log analysis showed that the course web site in general, and the 
lecture slides posted on the web site in particular, were popular and well-received. Transaction log analysis of 
lecture webcasts showed clearly that students used lecture webcasts primarily as a study tool and a supplement 
to in-person attendance. Attendance data indicate that, although students valued lectures, they frequently opted 
out of attending them. 

We suspect that the positive reception of the Chemistry 1 A course and the associated technology enhancements 
is related to a number of factors: 

• The enhancements were minimally disruptive to the teaching style and pedagogy of the teaching staff; 

• The enhancements increased convenience for both students and faculty; 

• The enhancements were “generic” and pedagogically neutral enough that students could use them flexibly 
and on their own terms (e.g., as a “safety net,” reviewing lectures online for exam study and replacement 
of missed lectures, repetition of difficult sections by non-native English speakers, downloading lecture 
slides for preparation and review, taking quizzes multiple times); 

• The overall quality of this large lecture course, as with many others on the UC Berkeley campus, is 
exceptionally high. The instructors in charge are dedicated to providing the best experience possible for 
students, and are constantly integrating student and TA feedback into course improvements; and 
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• The instructors are charismatic lecturers who can make a large lecture hall intimate through a number of 
devices that encourage student participation in lectures. 

C. The Implications for Sharing, Reuse, and University Culture 

Campus culture will have a significant impact on the likelihood that online teaching materials will be shared and 
reused by other faculty (Harley 2002). Our findings suggest that some cost-savings could be realized under 
certain circumstances, which may or may not carry over from semester to semester at UC Berkeley or other 
campuses that pride themselves on having active research faculty teach introductory courses. For example, 
although the campus is in theory supportive of “introductory course redesign,” our knowledge of administrator 
and faculty attitudes about educational technologies paint a picture of a research university community that is 
not yet ready to embrace the reuse of space and time in a systematic way. Interviews indicated that faculty and 
administrators at various levels of the campus were unaware of the potential cost savings in space and time that 
might be possible through the careful use of educational technologies. 

Moreover, our knowledge of Chemistry 1 A faculty behavior suggests that the successful wholesale adoption of 
technology enhancements from one semester to the next cannot be assumed. In reality, the sharing of teaching 
materials in a research university environment may be complicated by multiple factors such as faculty 
idiosyncrasies and the continuity of underlying support structures for technology enhancements. Replicating 
support mechanisms and customizing materials to one’s own course require investments of time and energy by 
teaching staff. We should note that the experience at UC Berkeley is probably not directly comparable to 
institutions where non-research, non-tenured faculty are responsible for teaching large introductory courses. In 
fact, the sharing of electronic teaching materials among faculty may occur more readily in institutions where 
introductory course curricula are standardized and where research faculty cede course development and delivery 
to lecturers or adjuncts. 

Given a change in campus culture and thinking, there is certainly the possibility that several instructors, or even 
instructors on other UC campuses, might be able to share online lecture materials. Hypothetically, the 
availability of a variety of online materials to every Chemistry 1A instructor could eliminate the need for 
‘reinventing’ the course and thus allow time-savings in preparing, organizing, and updating the course materials. 
Additionally, a rethinking of the time faculty devote to repetition of the same lectures multiple times in a week 
could potentially free instructors to creatively use the lecture time as a more student-interactive experience 
and/or reallocate space for other purposes. This rethinking seems particularly relevant given that students have 
independently found ways to integrate technology enhancements into their time management and study 
strategies. 

Finally, we suspect that any large scaling benefits will come either (1) when newly hired faculty, who might be 
more adroit with new technologies, enter the department, (2) if the course can be “modular” so that faculty can 
select materials that fit their learning goals, should their learning goals differ from the developers’ intentions, 
and/or (3) if the materials can be made available to off-site student populations at other institutions. 
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Appendix A: List of Activities Involved in Chemistry 1A 



D e velop m ent/Revisio n 


Preparing lecture graphics and overheads 
Creating and updating lab manual 
Revising discussion handbook 

Preparing and programming questions for on-line quizzes and assignments 
Designing and preparing of course website 


Delivery: Lectures 


Delivering the lecture, including demos and in-class video display 
Delivering the lecture 

Preparing and performing chemistry demonstrations 
Originating and displaying lecture hall video 
Encoding, streaming, archiving lecture webcasts 
Preparing for lecture (instructor) 

Setting up lecture browser presentation 


Delivery: Labs 


Supplying chemicals for experiments 
Attending TA meetings and lectures 

Attending weekly TA meetings 
Attending lectures 

Attending daily pre-laboratory check-in meetings 
Teaching and administering lab sections 
Preparing for laboratory teaching (TAs) 

Recruiting, training, and monitoring teaching staff 
Recruiting teaching staff 
TA training 
Monitoring labs 


Delivery: Quizzes, Exams, and Grading 


Grading homework quizzes, check-in quizzes, and pre-lab exercises 
Grading lab write-ups 

Creating homework quizzes (traditional version) 

Grading (instructor) 

Setting up and maintaining on-line pre-lab exercises and quizzes 
Setting up on-line quizzes and assignments 
Maintaining on-line quizzes and assignments 
Writing, proctoring, and grading exams 

Writing exams (Head TAs and instructors) 

Proctoring midterms 

Grading midterms 

Proctoring final exam 

Grading final exam 


Delivery: General 


Course-level administration and scheduling 

Pre-course lecture and lab scheduling 
Faculty-staff weekly meeting 
Course administrative duties 
Administration 
Office hours (instructor) 

Holding office hours and communicating with students 
Communicating and meeting with TAs 
Maintaining and hosting website 

Maintaining website content 
Hosting website 

Tutoring and other out-of-class student support 
Out-of-class teaching activities 

Preparing for and teaching scholars sections 
Office hours (TAs) 

Weekly review sessions 
Other out-of-class teaching activities 
Student enrollment, account, and database management 
Managing student enrollment 
Managing on-line accounts 
Setting-up student grades database 
Maintaining student grades database 
Updating, editing, and publishing the syllabus 
Updating the syllabus 
Editing and publishing the syllabus 
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Appendix B: Breakdown of Coursework and Total Course Points 







Year 1 Points 


Year 2 Points 


Final Exam points (total) 




350 


350 


Mid-terms 1, 2 & 3 (combined total) 




400 


375 A 


Laboratory Points (total) 




200 


198 


Homework Quiz Points (10 each) 




30 B 


35 c 


Check-in Question/Discussion 




20° 


14 E 


Pre-lab Assignments 




** 


27 b 


Lab Write-ups 




** 


** 


Total Points 




1,000 


999 


Extra Credit Points Possible (estimated) 




9 


9 


Grading Scale 


A = 


870- 1,000 


900- 1,000 




B = 


730 - 869 


750 - 899 




C = 


470 - 729 


500 - 749 




D = 


350-469 


350 -500 



The two best midterms were worth 150 points each, but lowest midterm score was weighted less (75 points). 
3 points each 
2.5 points each 

2 points each x 1 0 assignments 
1 point each x 14 assignments 
Points included in laboratory points total 
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Appendix C: Selected Student Survey Responses, Compared by Year 





Y1 On-line 






Y1 In-class 


Y1 In-class 








Post 


Y1 On-line 


Y1 On-line 


Post 


Post 


Y1 In-class 






Treatment 


Post Control 


Post 


Treatment 


Control 


Post 


Y2 Post 


Question 


Group 


Group 


All 


Group 


Group 


All 


All 




N= 69 


N=174 


N= 243 


N=219 


N=685 


N=904 


N=643 


How often do you visit the Chem 1A course 
website, or other Chem 1A on-line resources? 
















3+ times per week 


83% 


73% 


76% 


67% 


54% 


57% 


85% 


1-2 times per week 


15% 


24% 


21% 


31% 


32% 


32% 


14% 


less than once per week 


2% 


3% 


2% 


2% 


13% 


10% 


1% 


never 








1% 


0% 


0% 


1% 


How often do you watch lecture webcasts? 
















3+ times per week 


3% 


9% 


7% 


9% 


7% 


8% 


9% 


1-2 times per week 


23% 


29% 


30% 


23% 


21% 


21% 


19% 


less than once per week 


53% 


43% 


46% 


46% 


45% 


45% 


50% 


never 


21% 


19% 


19% 


22% 


27% 


26% 


23% 


I watch Fall 2001 Chem 1A lecture webcasts for 
















the following reasons: (check all that apply) 

Instead of attending live lecture 


25% 


27% 


27% 








26% 


To study and review material prior to exams 


56% 


65% 


63% 








41% 


After I miss a lecture 


69% 


65% 


66% 








72% 


Instead of reviewing notes/handouts 


10% 


10% 


10% 








n/a 


In addition to reviewing notes/handouts 


n/a 


n/a 


n/a 








31% 


Instead of seeking assistance from a GSI or professor 


6% 


15% 


13% 








8% 


To prepare for lab 


6% 


3% 


4% 










After attending a live lecture 


12% 


16% 


15% 










Other 


7% 


7% 


7% 








5% 


I would be willing to watch lecture webcasts 
entirely on-line instead of going to the lecture hall. 
















Yes 








16% 


17% 


17% 


14% 


No 








83% 


82% 


82% 


86% 


Both 








1% 


1% 


1% 




Did completing the on-line homework quizzes or 
pre-lab assignments in advance help you perform 
the tasks required in lab? 
















Yes 


70% 


n/a 


70% 








52% 


No 


30% 


n/a 


30% 








48% 


Did taking [on-line homework] quizzes or pre-lab 
assignments multiple times help you learn concepts 
about Chem 1A? 
















Yes 


85% 


n/a 


85% 








71% 


No 


15% 


n/a 


15% 








29% 


Would you be willing to do more activities on-line 
(e.g. quizzes, homework, discussion) so that you 
could spend fewer hours in the lab? 
















Yes 


75% 


69% 


71% 


85% 








No 


25% 


31% 


29% 


15% 








Since Chem 1A began, how often have 
you...completed an on-line pre-lab assignment or 
homework quiz with a group of other students? 
















Always 


2% 


4% 


3% 








9% 


Most of the time 


2% 


10% 


7% 








15% 


Some of the time 


21% 


55% 


45% 








34% 


Never 


76% 


31% 


45% 








42% 


Since Chem 1A began, how often have 
you...studied with other students for quizzes and/or 
examinations for Chem 1A? 
















Always 


10% 


14% 


13% 








18% 


Most of the time 


19% 


27% 


25% 








25% 


Some of the time 


37% 


36% 


36% 








39% 


Never 


33% 


23% 


26% 








18% 
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Since Chem 1A began, how often have you. ..used 
email to ask questions of your GSI or professor 
instead of going to office hours in person? 

Always 
Most of the time 
Some of the time 
Never 


14% 

23% 

38% 

26% 


11% 12% 

16% 18% 

40% 39% 

33% 31% 






6% 

13% 

43% 

39% 


On average how many hours PER WEEK have 
you spent in activities related to lecture for 
Chem 1A. 

Total (average) hours per week 
Standard Deviation 


7:02 

3:53 


7:22 7:16 

5:16 4:55 






6:55 

4:33 


On average how many hours PER WEEK have 
you spent in activities related to lab for Chem fA. 

Total (average) hours per week 
Standard Deviation 


7:53 

2:37 


7:34 7:40 

3:01 2:54 






6:54 

2:48 


In general, did you have difficulty completing lab 
experiments in the time allotted? 

Yes 

No 










13% 

87% 


How do you most often access Chem f A on-line 
materials through the Internet? (Choose one) 

Phone modem 
Cable modem 
DSL 

Campus ethemet 
Other 


15% 

3% 

10% 

72% 


3% 7% 

4% 4% 

5% 7% 

87% 82% 

1% 1% 


10% 

5% 

5% 

83% 

1% 


8% 8% 

4% 4% 

3% 3% 

86% 86% 

0% 


6% 

5% 

7% 

82% 

1% 


Did your experience with the technologies offered 
in Chem 1A affect your overall satisfaction with 
the course? 

Yes 

No 










87% 

13% 


I would recommend that other students take 
courses like Chem 1A that use: (check all that 
apply) 

Course websites 
Lecture slides 

On-line quizzes or assignments 

Lecture webcasts (video broadcast over the internet) 

Lecture browser (archived lecture webcasts w / search 

function) 

Accessing grades on the web 
On-line office hours 


100% 

88% 

87% 

76% 

94% 


95% 97% 

88% 88% 

56% 65% 

75% 75% 

90% 91% 

n/a 
n/a 






94% 

83% 

67% 

80% 

44% 

92% 

41% 


Do you have a learning disability, or are you 
physically challenged or differently abled? 

Yes 

No 


8% 

92% 


3% 5% 

97% 95% 






1% 

99% 


Is English your first language? 

Yes 

No 


68% 

32% 


44% 72% 

26% 28% 






78% 

22% 
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